KVM: x86: Clean up MSR interception code
Sean Christopherson
maintainercover KVM: x86: Clean up MSR interception code
Clean up KVM's MSR interception code (especially the SVM code, which is all kinds of ugly). The main goals are to:
- Make the SVM and VMX APIs consistent (and sane; the current SVM APIs have inverted polarity).
- Eliminate the shadow bitmaps that are used to determine intercepts on userspace MSR filter update.
Folks that are explicitly Cc'd, my plan/hope is to apply this in advance of landing the CET virtualization and mediated PMU series, so that we don't need to deal with extended the shadow bitmaps. Any reviews/testing you can provide to help make that happen would be greatly appreciated.
Note, this is a spiritual successor to the "Unify MSR intercepts in x86" series that was posted last year[*], but I started the versioning back at v1 as very, very little of the code actually survived, and there's obviously no true unification in this series. That series also had several bugs (that were never pointed out on list), so I wanted to make a clean break.
FWIW, I still like the _idea_ of unified code, but with the shadow bitmaps gone, it's not actually that much code, and the logic isn't all that complex. In the end, I couldn't convince myself that unifying that small amount of logic was worth taking on the complexity of generating and passing around bit numbers and bitmap pointers to common code (or adding 4 more kvm_x86_ops hooks).
[*] https://lore.kernel.org/kvm/[email protected]
Sean Christopherson (28):
KVM: SVM: Don't BUG if setting up the MSR intercept bitmaps fails
KVM: SVM: Tag MSR bitmap initialization helpers with __init
KVM: SVM: Use ARRAY_SIZE() to iterate over direct_access_msrs
KVM: SVM: Kill the VM instead of the host if MSR interception is buggy
KVM: x86: Use non-atomic bit ops to manipulate "shadow" MSR intercepts
KVM: SVM: Massage name and param of helper that merges vmcb01 and
vmcb12 MSRPMs
KVM: SVM: Clean up macros related to architectural MSRPM definitions
KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1
bitmaps
KVM: nSVM: Omit SEV-ES specific passthrough MSRs from L0+L1 bitmap
merge
KVM: nSVM: Don't initialize vmcb02 MSRPM with vmcb01's "always
passthrough"
KVM: SVM: Add helpers for accessing MSR bitmap that don't rely on
offsets
KVM: SVM: Implement and adopt VMX style MSR intercepts APIs
KVM: SVM: Pass through GHCB MSR if and only if VM is an SEV-ES guest
KVM: SVM: Drop "always" flag from list of possible passthrough MSRs
KVM: x86: Move definition of X2APIC_MSR() to lapic.h
KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter
change
KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter
change
KVM: x86: Rename msr_filter_changed() => recalc_msr_intercepts()
KVM: SVM: Rename init_vmcb_after_set_cpuid() to make it intercepts
specific
KVM: SVM: Fold svm_vcpu_init_msrpm() into its sole caller
KVM: SVM: Merge "after set CPUID" intercept recalc helpers
KVM: SVM: Drop explicit check on MSRPM offset when emulating SEV-ES
accesses
KVM: SVM: Move svm_msrpm_offset() to nested.c
KVM: SVM: Store MSRPM pointer as "void *" instead of "u32 *"
KVM: nSVM: Access MSRPM in 4-byte chunks only for merging L0 and L1
bitmaps
KVM: SVM: Return -EINVAL instead of MSR_INVALID to signal out-of-range
MSR
KVM: nSVM: Merge MSRPM in 64-bit chunks on 64-bit kernels
KVM: selftests: Verify KVM disable interception (for userspace) on
filter change
arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/lapic.h | 2 + arch/x86/kvm/svm/nested.c | 128 +++-- arch/x86/kvm/svm/sev.c | 29 +- arch/x86/kvm/svm/svm.c | 449 ++++++------------ arch/x86/kvm/svm/svm.h | 107 ++++- arch/x86/kvm/vmx/main.c | 6 +- arch/x86/kvm/vmx/vmx.c | 179 ++----- arch/x86/kvm/vmx/vmx.h | 9 - arch/x86/kvm/vmx/x86_ops.h | 2 +- arch/x86/kvm/x86.c | 8 +- .../kvm/x86/userspace_msr_exit_test.c | 8 + 13 files changed, 408 insertions(+), 523 deletions(-)
base-commit: 3f7b307757ecffc1c18ede9ee3cf9ce8101f3cc9
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 1/28 KVM: SVM: Don't BUG if setting up the MSR intercept bitmaps fails
WARN and reject module loading if there is a problem with KVM's MSR interception bitmaps. Panicking the host in this situation is inexcusable since it is trivially easy to propagate the error up the stack.
--- arch/x86/kvm/svm/svm.c | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-)
arch/x86/kvm/svm/svm.c +15 -12
@@ -945,7 +945,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu) } } -static void add_msr_offset (u32 offset) +static int add_msr_offset (u32 offset) {int i;
@@ -953,7 +953,7 @@ static void add_msr_offset(u32 offset)/* Offset already in list? */ if (msrpm_offsets[i] == offset) -return ; +return 0 ;/* Slot used by another offset? */ if (msrpm_offsets[i] !=MSR_INVALID )
@@ -962,17 +962,13 @@ static void add_msr_offset(u32 offset)/* Add offset to list */ msrpm_offsets[i] = offset; -return ; +return 0 ; } -/* -* If this BUG triggers the msrpm_offsets table has an overflow. Just -* increase MSRPM_OFFSETS in this case. -*/ -BUG (); +return -EIO ; } -static void init_msrpm_offsets (void ) +static int init_msrpm_offsets (void ) {int i;
@@ -982,10 +978,13 @@ static void init_msrpm_offsets(void)u32 offset; offset =svm_msrpm_offset (direct_access_msrs[i].index); -BUG_ON (offset ==MSR_INVALID ); +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; -add_msr_offset (offset); +if (WARN_ON_ONCE (add_msr_offset (offset))) +return -EIO ; } +return 0 ; }void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb)
@@ -5511,7 +5510,11 @@ static __init int svm_hardware_setup(void)memset (iopm_va,0xff ,PAGE_SIZE * (1 << order)); iopm_base =__sme_page_pa (iopm_pages); -init_msrpm_offsets (); + r =init_msrpm_offsets (); +if (r) { +__free_pages (__sme_pa_to_page (iopm_base),get_order (IOPM_SIZE )); +return r; + } kvm_caps.supported_xcr0 &= ~(XFEATURE_MASK_BNDREGS |XFEATURE_MASK_BNDCSR );
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 01/28] KVM: SVM: Don't BUG if setting up the MSR intercept bitmaps fails
Quoted 35 lines · Sean Christopherson
WARN and reject module loading if there is a problem with KVM's MSR interception bitmaps. Panicking the host in this situation is inexcusable since it is trivially easy to propagate the error up the stack.
Signed-off-by Sean Christopherson <[email protected]>---
arch/x86/kvm/svm/svm.c| 27 +++++++++++++++------------1 file changed, 15 insertions(+), 12 deletions(-)arch/x86/kvm/svm/svm.c +2
-2@@ -945,7 +945,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu) } } -static void add_msr_offset (u32 offset) +static int add_msr_offset (u32 offset) {int i;@@ -953,7 +953,7 @@ static void add_msr_offset(u32 offset)/* Offset already in list? */ if (msrpm_offsets[i] == offset) -return ; +return 0 ;/* Slot used by another offset? */ if (msrpm_offsets[i] !=MSR_INVALID )@@ -962,17 +962,13 @@ static void add_msr_offset(u32 offset)/* Add offset to list */ msrpm_offsets[i] = offset;
arch/x86/kvm/svm/svm.c static void add_msr_offset(u32 offset) · line 968
Quoted 10 lines
arch/x86/kvm/svm/svm.c +2
-6-return ; +return 0 ; } -/* -* If this BUG triggers the msrpm_offsets table has an overflow. Just -* increase MSRPM_OFFSETS in this case. -*/ -BUG (); +return -EIO ;
Would -ENOSPC be more appropriate here? And, instead of returning an integer, using a boolean might be better since the error code isn't propagated upwards.
Quoted 24 lines
arch/x86/kvm/svm/svm.c +6
-3} -static void init_msrpm_offsets (void ) +static int init_msrpm_offsets (void ) {int i;@@ -982,10 +978,13 @@ static void init_msrpm_offsets(void)u32 offset; offset =svm_msrpm_offset (direct_access_msrs[i].index); -BUG_ON (offset ==MSR_INVALID ); +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; -add_msr_offset (offset); +if (WARN_ON_ONCE (add_msr_offset (offset))) +return -EIO ; } +return 0 ; }void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb)
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5515
Quoted 7 lines
arch/x86/kvm/svm/svm.c +3
-1@@ -5511,7 +5510,11 @@ static __init int svm_hardware_setup(void)memset (iopm_va,0xff ,PAGE_SIZE * (1 << order)); iopm_base =__sme_page_pa (iopm_pages); -init_msrpm_offsets (); + r =init_msrpm_offsets (); +if (r) { +__free_pages (__sme_pa_to_page (iopm_base),get_order (IOPM_SIZE ));
__free_pages(iopm_pages, order);
And we can move init_msrpm_offsets() above the allocation of iopm_pages to avoid the need for rewinding. But I don't have a strong opinion on this, as it goes beyond a simple change to the return type.
Quoted 8 lines
arch/x86/kvm/svm/svm.c +2
-0+return r; + } kvm_caps.supported_xcr0 &= ~(XFEATURE_MASK_BNDREGS |XFEATURE_MASK_BNDCSR );Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerRe: [PATCH 01/28] KVM: SVM: Don't BUG if setting up the MSR intercept bitmaps fails
Quoted 40 lines · Chao Gao
Quoted 39 lines · Sean Christopherson
WARN and reject module loading if there is a problem with KVM's MSR interception bitmaps. Panicking the host in this situation is inexcusable since it is trivially easy to propagate the error up the stack.
Signed-off-by Sean Christopherson <[email protected]>---
arch/x86/kvm/svm/svm.c| 27 +++++++++++++++------------1 file changed, 15 insertions(+), 12 deletions(-)arch/x86/kvm/svm/svm.c +3
-3@@ -945,7 +945,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu) } } -static void add_msr_offset (u32 offset) +static int add_msr_offset (u32 offset) {int i;@@ -953,7 +953,7 @@ static void add_msr_offset(u32 offset)/* Offset already in list? */ if (msrpm_offsets[i] == offset) -return ; +return 0 ;/* Slot used by another offset? */ if (msrpm_offsets[i] !=MSR_INVALID )@@ -962,17 +962,13 @@ static void add_msr_offset(u32 offset)/* Add offset to list */ msrpm_offsets[i] = offset; -return ; +return 0 ; }
Quoted 8 lines
arch/x86/kvm/svm/svm.c static void add_msr_offset(u32 offset) · line 968
Quoted 6 lines
arch/x86/kvm/svm/svm.c +1
-5-/* -* If this BUG triggers the msrpm_offsets table has an overflow. Just -* increase MSRPM_OFFSETS in this case. -*/ -BUG (); +return -EIO ;Would
-ENOSPCbe more appropriate here?
Hmm, yeah. IIRC, I initially had -ENOSPC, but switched to -EIO to be consistent with how KVM typically reports its internal issues during module load. But as you point out, the error code isn't propagated up the stack.
Quoted 2 lines
And, instead of returning an integer, using a boolean might be better since the error code isn't propagated upwards.
I strongly prefer to return 0/-errno in any function that isn't a clear cut predicate, e.g. "return -EIO" is very obviously an error path (and -ENOSPC is even better), whereas understanding "return false" requires reading the rest of the function (and IMO this code isn't all that intuitive).
Quoted 27 lines
Quoted 27 lines
arch/x86/kvm/svm/svm.c +6
-3} -static void init_msrpm_offsets (void ) +static int init_msrpm_offsets (void ) {int i;@@ -982,10 +978,13 @@ static void init_msrpm_offsets(void)u32 offset; offset =svm_msrpm_offset (direct_access_msrs[i].index); -BUG_ON (offset ==MSR_INVALID ); +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; -add_msr_offset (offset); +if (WARN_ON_ONCE (add_msr_offset (offset))) +return -EIO ; } +return 0 ; }void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb)@@ -5511,7 +5510,11 @@ static __init int svm_hardware_setup(void)memset (iopm_va,0xff ,PAGE_SIZE * (1 << order)); iopm_base =__sme_page_pa (iopm_pages);
Quoted 6 lines
arch/x86/kvm/svm/svm.c static __init int svm_hardware_setup(void) · line 5515
Quoted 4 lines
arch/x86/kvm/svm/svm.c +3
-1-init_msrpm_offsets (); + r =init_msrpm_offsets (); +if (r) { +__free_pages (__sme_pa_to_page (iopm_base),get_order (IOPM_SIZE ));
__free_pages(iopm_pages, order);
Oh, yeah, good call.
Quoted 3 lines
And we can move
init_msrpm_offsets()above the allocation ofiopm_pagesto avoid the need for rewinding. But I don't have a strong opinion on this, as it goes beyond a simple change to the return type.
I considered that too. I decided not to bother since this code goes away by the end of the series. I thought about skipping this patch entirely, but I kept it since it should be easy to backport, e.g. if someone wants make their tree more developer friendly, and on the off chance the patches later in the series need to be droppped or reverted.
Sean Christopherson
maintainerPATCH 2/28 KVM: SVM: Tag MSR bitmap initialization helpers with __init
In reply to Sean ChristophersonTag init_msrpm_offsets() and add_msr_offset() with __init, as they're used only during hardware setup to map potential passthrough MSRs to offsets in the bitmap.
--- arch/x86/kvm/svm/svm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +2 -2
@@ -945,7 +945,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu) } } -static int add_msr_offset (u32 offset) +static __init int add_msr_offset (u32 offset) {int i;
@@ -968,7 +968,7 @@ static int add_msr_offset(u32 offset)return -EIO ; } -static int init_msrpm_offsets (void ) +static __init int init_msrpm_offsets (void ) {int i;
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 02/28] KVM: SVM: Tag MSR bitmap initialization helpers with __init
Quoted 5 lines · Sean Christopherson
Tag
init_msrpm_offsets()andadd_msr_offset()with__init, as they're used only during hardware setup to map potential passthrough MSRs to offsets in the bitmap.Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH 3/28 KVM: SVM: Use ARRAY_SIZE() to iterate over direct_access_msrs
In reply to Sean ChristophersonDrop the unnecessary and dangerous value-terminated behavior of direct_access_msrs, and simply iterate over the actual size of the array. The use in svm_set_x2apic_msr_interception() is especially sketchy, as it relies on unused capacity being zero-initialized, and '0' being outside the range of x2APIC MSRs.
To ensure the array and shadow_msr_intercept stay synchronized, simply assert that their sizes are identical (note the six 64-bit-only MSRs).
Note, direct_access_msrs will soon be removed entirely; keeping the assert synchronized with the array isn't expected to be along-term maintenance burden.
--- arch/x86/kvm/svm/svm.c | 18 +++++++++++------- arch/x86/kvm/svm/svm.h | 2 +- 2 files changed, 12 insertions(+), 8 deletions(-)
arch/x86/kvm/svm/svm.c +11 -7
@@ -86,7 +86,7 @@ static DEFINE_PER_CPU(u64, current_tsc_ratio);static const struct svm_direct_access_msrs {u32 index;/* Index of the MSR */ bool always;/* True if intercept is initially cleared */ -} direct_access_msrs[MAX_DIRECT_ACCESS_MSRS ] = { +} direct_access_msrs[] = { { .index =MSR_STAR , .always = true }, { .index =MSR_IA32_SYSENTER_CS , .always = true }, { .index =MSR_IA32_SYSENTER_EIP , .always = false },
@@ -144,9 +144,12 @@ static const struct svm_direct_access_msrs { { .index = X2APIC_MSR(APIC_TMICT), .always = false }, { .index = X2APIC_MSR(APIC_TMCCT), .always = false }, { .index = X2APIC_MSR(APIC_TDCR), .always = false }, - { .index = MSR_INVALID, .always = false }, }; +static_assert (ARRAY_SIZE (direct_access_msrs) == +MAX_DIRECT_ACCESS_MSRS -6 * !IS_ENABLED (CONFIG_X86_64 )); +#undef MAX_DIRECT_ACCESS_MSRS +/* * These 2 parameters are used to config the controls for Pause-Loop Exiting: * pause_filter_count: On processors that support Pause filtering(indicated
@@ -767,9 +770,10 @@ static int direct_access_msr_slot(u32 msr) {u32 i; -for (i =0 ; direct_access_msrs[i].index !=MSR_INVALID ; i++) +for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {if (direct_access_msrs[i].index == msr)return i; + }return -ENOENT ; }
@@ -891,7 +895,7 @@ void svm_vcpu_init_msrpm(struct kvm_vcpu *vcpu, u32 *msrpm) {int i; -for (i =0 ; direct_access_msrs[i].index !=MSR_INVALID ; i++) { +for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {if (!direct_access_msrs[i].always)continue ;set_msr_interception (vcpu, msrpm, direct_access_msrs[i].index,1 ,1 );
@@ -908,7 +912,7 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept)if (!x2avic_enabled)return ; -for (i =0 ; i <MAX_DIRECT_ACCESS_MSRS ; i++) { +for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {int index = direct_access_msrs[i].index; if ((index < APIC_BASE_MSR) ||
@@ -936,7 +940,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu)* will automatically get filtered through the MSR filter, so we are * back in sync after this. */ -for (i =0 ; direct_access_msrs[i].index !=MSR_INVALID ; i++) { +for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {u32 msr = direct_access_msrs[i].index;u32 read =test_bit (i, svm->shadow_msr_intercept.read);u32 write =test_bit (i, svm->shadow_msr_intercept.write);
@@ -974,7 +978,7 @@ static __init int init_msrpm_offsets(void)memset (msrpm_offsets,0xff ,sizeof (msrpm_offsets)); -for (i =0 ; direct_access_msrs[i].index !=MSR_INVALID ; i++) { +for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {u32 offset; offset =svm_msrpm_offset (direct_access_msrs[i].index);
arch/x86/kvm/svm/svm.h +1 -1
@@ -44,7 +44,7 @@ static inline struct page *__sme_pa_to_page(unsigned long pa)#define IOPM_SIZE PAGE_SIZE * 3#define MSRPM_SIZE PAGE_SIZE * 2 -#define MAX_DIRECT_ACCESS_MSRS 48 +#define MAX_DIRECT_ACCESS_MSRS 47#define MSRPM_OFFSETS 32extern u32 msrpm_offsets[MSRPM_OFFSETS ]__read_mostly ;extern bool npt_enabled;
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 03/28] KVM: SVM: Use ARRAY_SIZE() to iterate over direct_access_msrs
Quoted 6 lines · Sean Christopherson
Drop the unnecessary and dangerous value-terminated behavior of
direct_access_msrs, and simply iterate over the actual size of the array. The use insvm_set_x2apic_msr_interception()is especially sketchy, as it relies on unused capacity being zero-initialized, and '0' being outside the range of x2APIC MSRs.
Quoted 8 lines
To ensure the array and
shadow_msr_interceptstay synchronized, simply assert that their sizes are identical (note the six 64-bit-only MSRs).Note,
direct_access_msrswill soon be removed entirely; keeping the assert synchronized with the array isn't expected to be along-term maintenance burden.Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerPATCH 4/28 KVM: SVM: Kill the VM instead of the host if MSR interception is buggy
In reply to Sean ChristophersonWARN and kill the VM instead of panicking the host if KVM attempts to set or query MSR interception for an unsupported MSR. Accessing the MSR interception bitmaps only meaningfully affects post-VMRUN behavior, and KVM_BUG_ON() is guaranteed to prevent the current vCPU from doing VMRUN, i.e. there is no need to panic the entire host.
--- arch/x86/kvm/svm/svm.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/svm.c +4 -2
@@ -827,7 +827,8 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) bit_write =2 * (msr &0x0f ) +1 ; tmp = msrpm[offset]; -BUG_ON (offset ==MSR_INVALID ); +if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm)) +return false;return test_bit (bit_write, &tmp); }
@@ -858,7 +859,8 @@ static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm, bit_write =2 * (msr &0x0f ) +1 ; tmp = msrpm[offset]; -BUG_ON (offset ==MSR_INVALID ); +if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm)) +return ; read ?clear_bit (bit_read, &tmp) :set_bit (bit_read, &tmp); write ?clear_bit (bit_write, &tmp) :set_bit (bit_write, &tmp);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 04/28] KVM: SVM: Kill the VM instead of the host if MSR interception is buggy
Quoted 16 lines · Sean Christopherson
WARN and kill the VM instead of panicking the host if KVM attempts to set or query MSR interception for an unsupported MSR. Accessing the MSR interception bitmaps only meaningfully affects post-VMRUN behavior, and
KVM_BUG_ON()is guaranteed to prevent the current vCPU from doing VMRUN, i.e. there is no need to panic the entire host.Signed-off-by Sean Christopherson <[email protected]>---
arch/x86/kvm/svm/svm.c| 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)arch/x86/kvm/svm/svm.c +0
-0
arch/x86/kvm/svm/svm.c static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) · line 828
Quoted 2 lines
arch/x86/kvm/svm/svm.c +0
-0@@ -827,7 +827,8 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) bit_write =2 * (msr &0x0f ) +1 ; tmp = msrpm[offset];
not an issue with this patch. but shouldn't the offset be checked against MSR_INVALID before being used to index msrpm[]?
Quoted 20 lines
arch/x86/kvm/svm/svm.c +4
-2-BUG_ON (offset ==MSR_INVALID ); +if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm)) +return false;return test_bit (bit_write, &tmp); }@@ -858,7 +859,8 @@ static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm, bit_write =2 * (msr &0x0f ) +1 ; tmp = msrpm[offset]; -BUG_ON (offset ==MSR_INVALID ); +if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm)) +return ; read ?clear_bit (bit_read, &tmp) :set_bit (bit_read, &tmp); write ?clear_bit (bit_write, &tmp) :set_bit (bit_write, &tmp);Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerRe: [PATCH 04/28] KVM: SVM: Kill the VM instead of the host if MSR interception is buggy
Quoted 12 lines · Chao Gao
Quoted 11 lines · Sean Christopherson
WARN and kill the VM instead of panicking the host if KVM attempts to set or query MSR interception for an unsupported MSR. Accessing the MSR interception bitmaps only meaningfully affects post-VMRUN behavior, and
KVM_BUG_ON()is guaranteed to prevent the current vCPU from doing VMRUN, i.e. there is no need to panic the entire host.Signed-off-by Sean Christopherson <[email protected]>---
arch/x86/kvm/svm/svm.c| 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)
Quoted 10 lines
arch/x86/kvm/svm/svm.c static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) · line 828
Quoted 7 lines
arch/x86/kvm/svm/svm.c +0
-0@@ -827,7 +827,8 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) bit_write =2 * (msr &0x0f ) +1 ; tmp = msrpm[offset];not an issue with this patch. but shouldn't the offset be checked against
MSR_INVALIDbefore being used to index msrpm[]?
Oof, yes. To some extent, it _is_ a problem with this patch, because using KVM_BUG_ON() makes the OOB access less fatal. Though it's just a load, and code that should be unreachable, but still worth cleaning up.
Anyways, I'll place the KVM_BUG_ON()s in the right location as part of this patch.
Thanks!
Sean Christopherson
maintainerPATCH 5/28 KVM: x86: Use non-atomic bit ops to manipulate "shadow" MSR intercepts
In reply to Sean ChristophersonManipulate the MSR bitmaps using non-atomic bit ops APIs (two underscores), as the bitmaps are per-vCPU and are only ever accessed while vcpu->mutex is held.
--- arch/x86/kvm/svm/svm.c | 12 ++++++------ arch/x86/kvm/vmx/vmx.c | 8 ++++---- 2 files changed, 10 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/svm.c +6 -6
@@ -789,14 +789,14 @@ static void set_shadow_msr_intercept(struct kvm_vcpu *vcpu, u32 msr, int read,/* Set the shadow bitmaps to the desired intercept states */ if (read) -set_bit (slot, svm->shadow_msr_intercept.read); +__set_bit (slot, svm->shadow_msr_intercept.read);else -clear_bit (slot, svm->shadow_msr_intercept.read); +__clear_bit (slot, svm->shadow_msr_intercept.read);if (write) -set_bit (slot, svm->shadow_msr_intercept.write); +__set_bit (slot, svm->shadow_msr_intercept.write);else -clear_bit (slot, svm->shadow_msr_intercept.write); +__clear_bit (slot, svm->shadow_msr_intercept.write); }static bool valid_msr_intercept (u32 index)
@@ -862,8 +862,8 @@ static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm,if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm))return ; - read ?clear_bit (bit_read, &tmp) :set_bit (bit_read, &tmp); - write ?clear_bit (bit_write, &tmp) :set_bit (bit_write, &tmp); + read ?__clear_bit (bit_read, &tmp) :__set_bit (bit_read, &tmp); + write ?__clear_bit (bit_write, &tmp) :__set_bit (bit_write, &tmp); msrpm[offset] = tmp;
arch/x86/kvm/vmx/vmx.c +4 -4
@@ -4029,9 +4029,9 @@ void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) idx =vmx_get_passthrough_msr_slot (msr);if (idx >=0 ) {if (type &MSR_TYPE_R ) -clear_bit (idx, vmx->shadow_msr_intercept.read); +__clear_bit (idx, vmx->shadow_msr_intercept.read);if (type &MSR_TYPE_W ) -clear_bit (idx, vmx->shadow_msr_intercept.write); +__clear_bit (idx, vmx->shadow_msr_intercept.write); } if ((type & MSR_TYPE_R) &&
@@ -4071,9 +4071,9 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) idx =vmx_get_passthrough_msr_slot (msr);if (idx >=0 ) {if (type &MSR_TYPE_R ) -set_bit (idx, vmx->shadow_msr_intercept.read); +__set_bit (idx, vmx->shadow_msr_intercept.read);if (type &MSR_TYPE_W ) -set_bit (idx, vmx->shadow_msr_intercept.write); +__set_bit (idx, vmx->shadow_msr_intercept.write); }if (type &MSR_TYPE_R )
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 05/28] KVM: x86: Use non-atomic bit ops to manipulate "shadow" MSR intercepts
Quoted 62 lines · Sean Christopherson
Manipulate the MSR bitmaps using non-atomic bit ops APIs (two underscores), as the bitmaps are per-vCPU and are only ever accessed while
vcpu->mutexis held.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/svm.c | 12 ++++++------ arch/x86/kvm/vmx/vmx.c | 8 ++++---- 2 files changed, 10 insertions(+), 10 deletions(-)arch/x86/kvm/svm/svm.c +6
-6@@ -789,14 +789,14 @@ static void set_shadow_msr_intercept(struct kvm_vcpu *vcpu, u32 msr, int read,/* Set the shadow bitmaps to the desired intercept states */ if (read) -set_bit (slot, svm->shadow_msr_intercept.read); +__set_bit (slot, svm->shadow_msr_intercept.read);else -clear_bit (slot, svm->shadow_msr_intercept.read); +__clear_bit (slot, svm->shadow_msr_intercept.read);if (write) -set_bit (slot, svm->shadow_msr_intercept.write); +__set_bit (slot, svm->shadow_msr_intercept.write);else -clear_bit (slot, svm->shadow_msr_intercept.write); +__clear_bit (slot, svm->shadow_msr_intercept.write); }static bool valid_msr_intercept (u32 index)@@ -862,8 +862,8 @@ static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm,if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm))return ; - read ?clear_bit (bit_read, &tmp) :set_bit (bit_read, &tmp); - write ?clear_bit (bit_write, &tmp) :set_bit (bit_write, &tmp); + read ?__clear_bit (bit_read, &tmp) :__set_bit (bit_read, &tmp); + write ?__clear_bit (bit_write, &tmp) :__set_bit (bit_write, &tmp); msrpm[offset] = tmp;arch/x86/kvm/vmx/vmx.c +2
-2@@ -4029,9 +4029,9 @@ void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) idx =vmx_get_passthrough_msr_slot (msr);if (idx >=0 ) {if (type &MSR_TYPE_R ) -clear_bit (idx, vmx->shadow_msr_intercept.read); +__clear_bit (idx, vmx->shadow_msr_intercept.read);if (type &MSR_TYPE_W ) -clear_bit (idx, vmx->shadow_msr_intercept.write); +__clear_bit (idx, vmx->shadow_msr_intercept.write); } if ((type & MSR_TYPE_R) &&@@ -4071,9 +4071,9 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) idx =vmx_get_passthrough_msr_slot (msr);
arch/x86/kvm/vmx/vmx.c void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) · line 4079
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +2
-2if (idx >=0 ) {if (type &MSR_TYPE_R ) -set_bit (idx, vmx->shadow_msr_intercept.read); +__set_bit (idx, vmx->shadow_msr_intercept.read);if (type &MSR_TYPE_W ) -set_bit (idx, vmx->shadow_msr_intercept.write); +__set_bit (idx, vmx->shadow_msr_intercept.write); }if (type &MSR_TYPE_R )
Sean Christopherson
maintainerPATCH 6/28 KVM: SVM: Massage name and param of helper that merges vmcb01 and vmcb12 MSRPMs
In reply to Sean ChristophersonRenam nested_svm_vmrun_msrpm() to nested_svm_merge_msrpm() to better capture its role, and opportunistically feed it @vcpu instead of @svm, as grabbing "svm" only to turn around and grab svm->vcpu is rather silly.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 15 +++++++-------- arch/x86/kvm/svm/svm.c | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/nested.c +7 -8
@@ -189,8 +189,9 @@ void recalc_intercepts(struct vcpu_svm *svm)* is optimized in that it only merges the parts where KVM MSR permission bitmap * may contain zero bits. */ -static bool nested_svm_vmrun_msrpm (struct vcpu_svm *svm) +static bool nested_svm_merge_msrpm (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu);int i;/*
@@ -205,7 +206,7 @@ static bool nested_svm_vmrun_msrpm(struct vcpu_svm *svm)if (!svm->nested.force_msr_bitmap_recalc) {struct hv_vmcb_enlightenments *hve = &svm->nested.ctl.hv_enlightenments; -if (kvm_hv_hypercall_enabled (&svm->vcpu) && +if (kvm_hv_hypercall_enabled (vcpu) && hve->hv_enlightenments_control.msr_bitmap && (svm->nested.ctl.clean &BIT (HV_VMCB_NESTED_ENLIGHTENMENTS ))) goto set_msrpm_base_pa;
@@ -230,7 +231,7 @@ static bool nested_svm_vmrun_msrpm(struct vcpu_svm *svm) offset = svm->nested.ctl.msrpm_base_pa + (p *4 ); -if (kvm_vcpu_read_guest (&svm->vcpu, offset, &value,4 )) +if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))return false; svm->nested.msrpm[p] = svm->msrpm[p] | value;
@@ -937,7 +938,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)if (enter_svm_guest_mode (vcpu, vmcb12_gpa, vmcb12, true)) goto out_exit_err; -if (nested_svm_vmrun_msrpm (svm)) +if (nested_svm_merge_msrpm (vcpu)) goto out; out_exit_err:
@@ -1819,13 +1820,11 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,static bool svm_get_nested_state_pages (struct kvm_vcpu *vcpu) { -struct vcpu_svm *svm =to_svm (vcpu); -if (WARN_ON (!is_guest_mode (vcpu)))return true;if (!vcpu->arch.pdptrs_from_userspace && - !nested_npt_enabled (svm) &&is_pae_paging (vcpu)) + !nested_npt_enabled (to_svm (vcpu)) &&is_pae_paging (vcpu))/* * Reload the guest's PDPTRs since after a migration * the guest CR3 might be restored prior to setting the nested
@@ -1834,7 +1833,7 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)if (CC (!load_pdptrs (vcpu, vcpu->arch.cr3)))return false; -if (!nested_svm_vmrun_msrpm (svm)) { +if (!nested_svm_merge_msrpm (vcpu)) { vcpu->run->exit_reason =KVM_EXIT_INTERNAL_ERROR ; vcpu->run->internal.suberror =KVM_INTERNAL_ERROR_EMULATION ;
arch/x86/kvm/svm/svm.c +1 -1
@@ -3134,7 +3134,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)* * For nested: * The handling of the MSR bitmap for L2 guests is done in -* nested_svm_vmrun_msrpm. +* nested_svm_merge_msrpm(). * We update the L1 MSR bit as well since it will end up * touching the MSR anyway now. */
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 06/28] KVM: SVM: Massage name and param of helper that merges vmcb01 and vmcb12 MSRPMs
Quoted 1 line · Sean Christopherson
Renam
nested_svm_vmrun_msrpm()tonested_svm_merge_msrpm()to better
"Renam" -> "Rename".
Quoted 90 lines
capture its role, and opportunistically feed it @vcpu instead of @svm, as grabbing "svm" only to turn around and grab
svm->vcpuis rather silly.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/nested.c | 15 +++++++-------- arch/x86/kvm/svm/svm.c | 2 +- 2 files changed, 8 insertions(+), 9 deletions(-)arch/x86/kvm/svm/nested.c +7
-8@@ -189,8 +189,9 @@ void recalc_intercepts(struct vcpu_svm *svm)* is optimized in that it only merges the parts where KVM MSR permission bitmap * may contain zero bits. */ -static bool nested_svm_vmrun_msrpm (struct vcpu_svm *svm) +static bool nested_svm_merge_msrpm (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu);int i;/* @@ -205,7 +206,7 @@ static bool nested_svm_vmrun_msrpm(struct vcpu_svm *svm)if (!svm->nested.force_msr_bitmap_recalc) {struct hv_vmcb_enlightenments *hve = &svm->nested.ctl.hv_enlightenments; -if (kvm_hv_hypercall_enabled (&svm->vcpu) && +if (kvm_hv_hypercall_enabled (vcpu) && hve->hv_enlightenments_control.msr_bitmap && (svm->nested.ctl.clean &BIT (HV_VMCB_NESTED_ENLIGHTENMENTS ))) goto set_msrpm_base_pa;@@ -230,7 +231,7 @@ static bool nested_svm_vmrun_msrpm(struct vcpu_svm *svm) offset = svm->nested.ctl.msrpm_base_pa + (p *4 ); -if (kvm_vcpu_read_guest (&svm->vcpu, offset, &value,4 )) +if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))return false; svm->nested.msrpm[p] = svm->msrpm[p] | value;@@ -937,7 +938,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)if (enter_svm_guest_mode (vcpu, vmcb12_gpa, vmcb12, true)) goto out_exit_err; -if (nested_svm_vmrun_msrpm (svm)) +if (nested_svm_merge_msrpm (vcpu)) goto out; out_exit_err:@@ -1819,13 +1820,11 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu,static bool svm_get_nested_state_pages (struct kvm_vcpu *vcpu) { -struct vcpu_svm *svm =to_svm (vcpu); -if (WARN_ON (!is_guest_mode (vcpu)))return true;if (!vcpu->arch.pdptrs_from_userspace && - !nested_npt_enabled (svm) &&is_pae_paging (vcpu)) + !nested_npt_enabled (to_svm (vcpu)) &&is_pae_paging (vcpu))/* * Reload the guest's PDPTRs since after a migration * the guest CR3 might be restored prior to setting the nested @@ -1834,7 +1833,7 @@ static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu)if (CC (!load_pdptrs (vcpu, vcpu->arch.cr3)))return false; -if (!nested_svm_vmrun_msrpm (svm)) { +if (!nested_svm_merge_msrpm (vcpu)) { vcpu->run->exit_reason =KVM_EXIT_INTERNAL_ERROR ; vcpu->run->internal.suberror =KVM_INTERNAL_ERROR_EMULATION ;arch/x86/kvm/svm/svm.c +1
-1@@ -3134,7 +3134,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)* * For nested: * The handling of the MSR bitmap for L2 guests is done in -* nested_svm_vmrun_msrpm. +* nested_svm_merge_msrpm(). * We update the L1 MSR bit as well since it will end up * touching the MSR anyway now. */
Sean Christopherson
maintainerPATCH 7/28 KVM: SVM: Clean up macros related to architectural MSRPM definitions
In reply to Sean ChristophersonMove SVM's MSR Permissions Map macros to svm.h in antipication of adding helpers that are available to SVM code, and opportunistically replace a variety of open-coded literals with (hopefully) informative macros.
Opportunistically open code ARRAY_SIZE(msrpm_ranges) instead of wrapping it as NUM_MSR_MAPS, which is an ambiguous name even if it were qualified with "SVM_MSRPM".
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 18 +++++++++--------- arch/x86/kvm/svm/svm.h | 17 ++++++++++++++++- 2 files changed, 25 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/svm.c +9 -9
@@ -266,24 +266,24 @@ static DEFINE_MUTEX(vmcb_dump_mutex);*/ static int tsc_aux_uret_slot__read_mostly =-1 ; -static const u32 msrpm_ranges[] = {0 ,0xc0000000 ,0xc0010000 }; - -#define NUM_MSR_MAPS ARRAY_SIZE(msrpm_ranges) -#define MSRS_RANGE_SIZE 2048 -#define MSRS_IN_RANGE (MSRS_RANGE_SIZE * 8 / 2) +static const u32 msrpm_ranges[] = { +SVM_MSRPM_RANGE_0_BASE_MSR , +SVM_MSRPM_RANGE_1_BASE_MSR , +SVM_MSRPM_RANGE_2_BASE_MSR +};u32 svm_msrpm_offset (u32 msr) {u32 offset;int i; -for (i =0 ; i <NUM_MSR_MAPS ; i++) { +for (i =0 ; i <ARRAY_SIZE (msrpm_ranges); i++) {if (msr < msrpm_ranges[i] || - msr >= msrpm_ranges[i] +MSRS_IN_RANGE ) + msr >= msrpm_ranges[i] +SVM_MSRS_PER_RANGE )continue ; - offset = (msr - msrpm_ranges[i]) /4 ;/* 4 msrs per u8 */ - offset += (i *MSRS_RANGE_SIZE );/* add range offset */ + offset = (msr - msrpm_ranges[i]) /SVM_MSRS_PER_BYTE ; + offset += (i *SVM_MSRPM_BYTES_PER_RANGE );/* add range offset */ /* Now we have the u8 offset - but need the u32 offset */ return offset /4 ;
arch/x86/kvm/svm/svm.h +16 -1
@@ -613,11 +613,26 @@ static inline void svm_vmgexit_no_action(struct vcpu_svm *svm, u64 data)svm_vmgexit_set_return_code (svm,GHCB_HV_RESP_NO_ACTION , data); } -/* svm.c */ +/* +* The MSRPM is 8KiB in size, divided into four 2KiB ranges (the fourth range +* is reserved). Each MSR within a range is covered by two bits, one each for +* read (bit 0) and write (bit 1), where a bit value of '1' means intercepted. +*/ +#define SVM_MSRPM_BYTES_PER_RANGE 2048 +#define SVM_BITS_PER_MSR 2 +#define SVM_MSRS_PER_BYTE (BITS_PER_BYTE / SVM_BITS_PER_MSR) +#define SVM_MSRS_PER_RANGE (SVM_MSRPM_BYTES_PER_RANGE * SVM_MSRS_PER_BYTE) +static_assert (SVM_MSRS_PER_RANGE ==8192 ); + +#define SVM_MSRPM_RANGE_0_BASE_MSR 0 +#define SVM_MSRPM_RANGE_1_BASE_MSR 0xc0000000 +#define SVM_MSRPM_RANGE_2_BASE_MSR 0xc0010000 +#define MSR_INVALID 0xffffffffU#define DEBUGCTL_RESERVED_BITS (~DEBUGCTLMSR_LBR) +/* svm.c */ extern bool dump_invalid_vmcb;u32 svm_msrpm_offset (u32 msr);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 8/28 KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
In reply to Sean ChristophersonUse a dedicated array of MSRPM offsets to merge L0 and L1 bitmaps, i.e. to merge KVM's vmcb01 bitmap with L1's vmcb12 bitmap. This will eventually allow for the removal of direct_access_msrs, as the only path where tracking the offsets is truly justified is the merge for nested SVM, where merging in chunks is an easy way to batch uaccess reads/writes.
Opportunistically omit the x2APIC MSRs from the merge-specific array instead of filtering them out at runtime.
Note, disabling interception of XSS, EFER, PAT, GHCB, and TSC_AUX is mutually exclusive with nested virtualization, as KVM passes through the MSRs only for SEV-ES guests, and KVM doesn't support nested virtualization for SEV+ guests. Defer removing those MSRs to a future cleanup in order to make this refactoring as benign as possible.
--- arch/x86/kvm/svm/nested.c | 72 +++++++++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 4 +++ arch/x86/kvm/svm/svm.h | 2 ++ 3 files changed, 67 insertions(+), 11 deletions(-)
arch/x86/kvm/svm/nested.c +61 -11
@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ; +static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = { +MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , + #ifdef CONFIG_X86_64 +MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , +MSR_CSTAR , +MSR_SYSCALL_MASK , + #endif +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , +MSR_IA32_FLUSH_CMD , +MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , + +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , +MSR_TSC_AUX , + }; +int i, j; + +for (i =0 ; i <ARRAY_SIZE (merge_msrs); i++) { +u32 offset =svm_msrpm_offset (merge_msrs[i]); + +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; + +for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) { +if (nested_svm_msrpm_merge_offsets[j] == offset) +break ; + } + +if (j < nested_svm_nr_msrpm_merge_offsets) +continue ; + +if (WARN_ON (j >=ARRAY_SIZE (nested_svm_msrpm_merge_offsets))) +return -EIO ; + + nested_svm_msrpm_merge_offsets[j] = offset; + nested_svm_nr_msrpm_merge_offsets++; + } + +return 0 ; +} +/* * Merge L0's (KVM) and L1's (Nested VMCB) MSR permission bitmaps. The function * is optimized in that it only merges the parts where KVM MSR permission bitmap
@@ -216,19 +274,11 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return true; -for (i =0 ; i <MSRPM_OFFSETS ; i++) { -u32 value, p; +for (i =0 ; i < nested_svm_nr_msrpm_merge_offsets; i++) { +const int p = nested_svm_msrpm_merge_offsets[i]; +u32 value;u64 offset; -if (msrpm_offsets[i] ==0xffffffff ) -break ; - - p = msrpm_offsets[i]; - -/* x2apic msrs are intercepted always for the nested guest */ -if (is_x2apic_msrpm_offset (p)) -continue ; - offset = svm->nested.ctl.msrpm_base_pa + (p *4 );if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))
arch/x86/kvm/svm/svm.c +4 -0
@@ -5689,6 +5689,10 @@ static int __init svm_init(void)if (!kvm_is_svm_supported ())return -EOPNOTSUPP ; + r =nested_svm_init_msrpm_merge_offsets (); +if (r) +return r; + r =kvm_x86_vendor_init (&svm_init_ops);if (r)return r;
arch/x86/kvm/svm/svm.h +2 -0
@@ -686,6 +686,8 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)return vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_NMI ); } +int __init nested_svm_init_msrpm_merge_offsets (void ); +int enter_svm_guest_mode (struct kvm_vcpu *vcpu,u64 vmcb_gpa,struct vmcb *vmcb12,bool from_vmrun);void svm_leave_nested (struct kvm_vcpu *vcpu);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
Quoted 27 lines · Sean Christopherson
Use a dedicated array of MSRPM offsets to merge L0 and L1 bitmaps, i.e. to merge KVM's vmcb01 bitmap with L1's vmcb12 bitmap. This will eventually allow for the removal of
direct_access_msrs, as the only path where tracking the offsets is truly justified is the merge for nested SVM, where merging in chunks is an easy way to batch uaccess reads/writes.Opportunistically omit the x2APIC MSRs from the merge-specific array instead of filtering them out at runtime.
Note, disabling interception of XSS, EFER, PAT, GHCB, and
TSC_AUXis mutually exclusive with nested virtualization, as KVM passes through the MSRs only for SEV-ES guests, and KVM doesn't support nested virtualization for SEV+ guests. Defer removing those MSRs to a future cleanup in order to make this refactoring as benign as possible.Signed-off-by Sean Christopherson <[email protected]>---
arch/x86/kvm/svm/nested.c | 72 +++++++++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 4 +++ arch/x86/kvm/svm/svm.h | 2 ++ 3 files changed, 67 insertions(+), 11 deletions(-)arch/x86/kvm/svm/nested.c +0
-0
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 187
Quoted 4 lines
arch/x86/kvm/svm/nested.c +1
-0@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ;
I understand how the array size (i.e., 9) was determined :). But, adding a comment explaining this would be quite helpful
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = { +MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , + #ifdef CONFIG_X86_64
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 207
Quoted 10 lines
arch/x86/kvm/svm/nested.c +10
-0+MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , +MSR_CSTAR , +MSR_SYSCALL_MASK , + #endif +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , + MSR_IA32_FLUSH_CMD,
MSR_IA32_DEBUGCTLMSR is missing, but it's benign since it shares the same offset as MSR_IA32_LAST* below.
I'm a bit concerned that we might overlook adding new MSRs to this array in the future, which could lead to tricky bugs. But I have no idea how to avoid this. Removing this array and iterating over direct_access_msrs[] directly is an option but it contradicts this series as one of its purposes is to remove direct_access_msrs[].
Quoted 20 lines
arch/x86/kvm/svm/nested.c +11
-0+MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , + +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , + MSR_TSC_AUX, + };if (
kvm_vcpu_read_guest(vcpu, offset, &value, 4))arch/x86/kvm/svm/svm.c +0
-0
arch/x86/kvm/svm/svm.c static int __init svm_init(void) · line 5695
Quoted 7 lines
arch/x86/kvm/svm/svm.c +4
-0@@ -5689,6 +5689,10 @@ static int __init svm_init(void)if (!kvm_is_svm_supported ())return -EOPNOTSUPP ; + r =nested_svm_init_msrpm_merge_offsets (); +if (r) +return r; +
If the offset array is used for nested virtualization only, how about guarding this with nested virtualization? For example, in svm_hardware_setup():
if (nested) {
r = nested_svm_init_msrpm_merge_offsets();
if (r)
goto err;
pr_info("Nested Virtualization enabled\n");
kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE);
}Quoted 19 lines
arch/x86/kvm/svm/svm.c +0
-0r =kvm_x86_vendor_init (&svm_init_ops);if (r)return r;arch/x86/kvm/svm/svm.h +2
-0@@ -686,6 +686,8 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)return vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_NMI ); } +int __init nested_svm_init_msrpm_merge_offsets (void ); +int enter_svm_guest_mode (struct kvm_vcpu *vcpu,u64 vmcb_gpa,struct vmcb *vmcb12,bool from_vmrun);void svm_leave_nested (struct kvm_vcpu *vcpu);Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerRe: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
Quoted 8 lines · Chao Gao
Quoted 8 lines
arch/x86/kvm/svm/nested.c +0
-0@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } }
Quoted 4 lines
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 187
Quoted 1 line
arch/x86/kvm/svm/nested.c +1
-0+static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ;I understand how the array size (i.e., 9) was determined :). But, adding a comment explaining this would be quite helpful
Yeah, I'll write a comment explaining what all is going on.
Quoted 15 lines
Quoted 15 lines
arch/x86/kvm/svm/nested.c +15
-0+static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = { +MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , + #ifdef CONFIG_X86_64 +MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , + MSR_CSTAR,
Quoted 8 lines
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 207
Quoted 5 lines
arch/x86/kvm/svm/nested.c +5
-0+MSR_SYSCALL_MASK , + #endif +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , + MSR_IA32_FLUSH_CMD,
MSR_IA32_DEBUGCTLMSRis missing, but it's benign since it shares the same offset as MSR_IA32_LAST* below.
Gah. Once all is said and done, it's not supposed to be in this list because its only passed through for SEV-ES guests, but my intent was to keep it in this patch, and then removeit along with XSS, EFER, PAT, GHCB, and TSC_AUX in the next.
This made me realize that merging in chunks has a novel flaw: if there is an MSR that KVM *doesn't* want to give L2 access to, then KVM needs to ensure its offset isn't processed, i.e. that there isn't a "collision" with another MSR. I don't think it's a major concern, because the x2APIC MSRs are nicely isolated, and off the top of my head I can't think of any MSRs that fall into that bucket. But it's something worth calling out in a comment, at least.
Quoted 2 lines
I'm a bit concerned that we might overlook adding new MSRs to this array in the future, which could lead to tricky bugs. But I have no idea how to avoid this.
Me either. One option would be to use wrapper macros for the interception helpers to fill an array at compile time (similar to how kernel exception fixup works), but (a) it requires modifications to the linker scripts to generate the arrays,
(b) is quite confusing/complex, and (c) it doesn't actually solve the problem, it just inverts the problem. Because as above, there are MSRs we *don't* want to expose to L2, and so we'd need to add yet more code to filter those out.
And the failure mode for the inverted case would be worse, because if we missed an MSR, KVM would incorrectly give L2 access to an MSR. Whereas with the current approach, a missed MSR simply means L2 gets a slower path; but functionally, it's fine (and it has to be fine, because KVM can't force L1 to disable interception).
Quoted 3 lines
Removing this array and iterating over direct_access_msrs[] directly is an option but it contradicts this series as one of its purposes is to remove direct_access_msrs[].
Using direct_access_msrs[] wouldn't solve the problem either, because nothing ensures _that_ array is up-to-date either.
The best idea I have is to add a test that verifies the MSRs that are supposed to be passed through actually are passed through. It's still effectively manual checking, but it would require us to screw up twice, i.e. forget to update both the array and the test. The problem is that there's no easy and foolproof way to verify that an MSR is passed through in a selftest.
E.g. it would be possible to precisely detect L2 => L0 MSR exits via a BPF program, but incorporating a BPF program into a KVM selftest just to detect exits isn't something I'm keen on doing (or maintaining).
Using the "exits" stat isn't foolproof due to NMIs (IRQs can be accounted for via "irq_exits", and to a lesser extent page faults (especially if shadow paging is in use).
If KVM provided an "msr_exits" stats, it would be trivial to verify interception via a selftest, but I can't quite convince myself that MSR exits are interesting enough to warrant their own stat.
Quoted 23 lines
Quoted 23 lines
arch/x86/kvm/svm/nested.c +11
-0+MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , + +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , + MSR_TSC_AUX, + };if (
kvm_vcpu_read_guest(vcpu, offset, &value, 4))arch/x86/kvm/svm/svm.c +0
-0@@ -5689,6 +5689,10 @@ static int __init svm_init(void)if (!kvm_is_svm_supported ())return -EOPNOTSUPP ;
Quoted 7 lines
arch/x86/kvm/svm/svm.c static int __init svm_init(void) · line 5695
Quoted 4 lines
arch/x86/kvm/svm/svm.c +4
-0+ r =nested_svm_init_msrpm_merge_offsets (); +if (r) +return r; +If the offset array is used for nested virtualization only, how about guarding this with nested virtualization? For example, in
svm_hardware_setup():
Good idea, I'll do that in the next version.
Quoted 29 lines
if (nested) { r = nested_svm_init_msrpm_merge_offsets(); if (r) goto err; pr_info("Nested Virtualization enabled\n"); kvm_enable_efer_bits(EFER_SVME | EFER_LMSLE); }Quoted 19 lines
arch/x86/kvm/svm/svm.c +0
-0r =kvm_x86_vendor_init (&svm_init_ops);if (r)return r;arch/x86/kvm/svm/svm.h +2
-0@@ -686,6 +686,8 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)return vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_NMI ); } +int __init nested_svm_init_msrpm_merge_offsets (void ); +int enter_svm_guest_mode (struct kvm_vcpu *vcpu,u64 vmcb_gpa,struct vmcb *vmcb12,bool from_vmrun);void svm_leave_nested (struct kvm_vcpu *vcpu);Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
Quoted 11 lines
Me either. One option would be to use wrapper macros for the interception helpers to fill an array at compile time (similar to how kernel exception fixup works), but (a) it requires modifications to the linker scripts to generate the arrays,
(b) is quite confusing/complex, and (c) it doesn't actually solve the problem, it just inverts the problem. Because as above, there are MSRs we *don't* want to expose to L2, and so we'd need to add yet more code to filter those out.And the failure mode for the inverted case would be worse, because if we missed an MSR, KVM would incorrectly give L2 access to an MSR. Whereas with the current approach, a missed MSR simply means L2 gets a slower path; but functionally, it's fine (and it has to be fine, because KVM can't force L1 to disable interception).
I agree. The risk of breaking functionality should be very low.
Quoted 18 lines
Quoted 4 lines
Removing this array and iterating over direct_access_msrs[] directly is an option but it contradicts this series as one of its purposes is to remove direct_access_msrs[].
Using direct_access_msrs[] wouldn't solve the problem either, because nothing ensures _that_ array is up-to-date either.
The best idea I have is to add a test that verifies the MSRs that are supposed to be passed through actually are passed through. It's still effectively manual checking, but it would require us to screw up twice, i.e. forget to update both the array and the test. The problem is that there's no easy and foolproof way to verify that an MSR is passed through in a selftest.
E.g. it would be possible to precisely detect L2 => L0 MSR exits via a BPF program, but incorporating a BPF program into a KVM selftest just to detect exits isn't something I'm keen on doing (or maintaining).
Quoted 7 lines
Using the "exits" stat isn't foolproof due to NMIs (IRQs can be accounted for via "
irq_exits", and to a lesser extent page faults (especially if shadow paging is in use).If KVM provided an "
msr_exits" stats, it would be trivial to verify interception via a selftest, but I can't quite convince myself that MSR exits are interesting enough to warrant their own stat.
Yes, there doesn't seem to be an easy solution to this problem. Given its low risk, I think we can live with it for now and revisit the issue if it truely becomes a recurring problem (i.e., people keep forgetting to update the array).
Paolo Bonzini
maintainerRe: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
In reply to Sean ChristophersonQuoted 27 lines · Sean Christopherson
Use a dedicated array of MSRPM offsets to merge L0 and L1 bitmaps, i.e. to merge KVM's vmcb01 bitmap with L1's vmcb12 bitmap. This will eventually allow for the removal of
direct_access_msrs, as the only path where tracking the offsets is truly justified is the merge for nested SVM, where merging in chunks is an easy way to batch uaccess reads/writes.Opportunistically omit the x2APIC MSRs from the merge-specific array instead of filtering them out at runtime.
Note, disabling interception of XSS, EFER, PAT, GHCB, and
TSC_AUXis mutually exclusive with nested virtualization, as KVM passes through the MSRs only for SEV-ES guests, and KVM doesn't support nested virtualization for SEV+ guests. Defer removing those MSRs to a future cleanup in order to make this refactoring as benign as possible.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/nested.c | 72 +++++++++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 4 +++ arch/x86/kvm/svm/svm.h | 2 ++ 3 files changed, 67 insertions(+), 11 deletions(-)arch/x86/kvm/svm/nested.c +0
-0
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 192
Quoted 9 lines
arch/x86/kvm/svm/nested.c +6
-0@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ; +static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = {
"static const", please.
Paolo
Quoted 106 lines
arch/x86/kvm/svm/nested.c +55
-11+MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , + #ifdef CONFIG_X86_64 +MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , +MSR_CSTAR , +MSR_SYSCALL_MASK , + #endif +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , +MSR_IA32_FLUSH_CMD , +MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , + +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , + MSR_TSC_AUX, + }; +int i, j; + +for (i =0 ; i <ARRAY_SIZE (merge_msrs); i++) { +u32 offset =svm_msrpm_offset (merge_msrs[i]); + +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; + +for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) { +if (nested_svm_msrpm_merge_offsets[j] == offset) +break ; + } + +if (j < nested_svm_nr_msrpm_merge_offsets) +continue ; + +if (WARN_ON (j >=ARRAY_SIZE (nested_svm_msrpm_merge_offsets))) +return -EIO ; + + nested_svm_msrpm_merge_offsets[j] = offset; + nested_svm_nr_msrpm_merge_offsets++; + } + +return 0 ; +} +/* * Merge L0's (KVM) and L1's (Nested VMCB) MSR permission bitmaps. The function * is optimized in that it only merges the parts where KVM MSR permission bitmap @@ -216,19 +274,11 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return true; -for (i =0 ; i <MSRPM_OFFSETS ; i++) { -u32 value, p; +for (i =0 ; i < nested_svm_nr_msrpm_merge_offsets; i++) { +const int p = nested_svm_msrpm_merge_offsets[i]; +u32 value;u64 offset; -if (msrpm_offsets[i] ==0xffffffff ) -break ; - - p = msrpm_offsets[i]; - -/* x2apic msrs are intercepted always for the nested guest */ -if (is_x2apic_msrpm_offset (p)) -continue ; - offset = svm->nested.ctl.msrpm_base_pa + (p *4 );if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))arch/x86/kvm/svm/svm.c +4
-0@@ -5689,6 +5689,10 @@ static int __init svm_init(void)if (!kvm_is_svm_supported ())return -EOPNOTSUPP ; + r =nested_svm_init_msrpm_merge_offsets (); +if (r) +return r; + r =kvm_x86_vendor_init (&svm_init_ops);if (r)return r;arch/x86/kvm/svm/svm.h +2
-0@@ -686,6 +686,8 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)return vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_NMI ); } +int __init nested_svm_init_msrpm_merge_offsets (void ); +int enter_svm_guest_mode (struct kvm_vcpu *vcpu,u64 vmcb_gpa,struct vmcb *vmcb12,bool from_vmrun);void svm_leave_nested (struct kvm_vcpu *vcpu);
Sean Christopherson
maintainerRe: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
Quoted 23 lines · Paolo Bonzini
Quoted 22 lines · Sean Christopherson
Use a dedicated array of MSRPM offsets to merge L0 and L1 bitmaps, i.e. to merge KVM's vmcb01 bitmap with L1's vmcb12 bitmap. This will eventually allow for the removal of
direct_access_msrs, as the only path where tracking the offsets is truly justified is the merge for nested SVM, where merging in chunks is an easy way to batch uaccess reads/writes.Opportunistically omit the x2APIC MSRs from the merge-specific array instead of filtering them out at runtime.
Note, disabling interception of XSS, EFER, PAT, GHCB, and
TSC_AUXis mutually exclusive with nested virtualization, as KVM passes through the MSRs only for SEV-ES guests, and KVM doesn't support nested virtualization for SEV+ guests. Defer removing those MSRs to a future cleanup in order to make this refactoring as benign as possible.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/nested.c | 72 +++++++++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 4 +++ arch/x86/kvm/svm/svm.h | 2 ++ 3 files changed, 67 insertions(+), 11 deletions(-)
Quoted 15 lines
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 191
Quoted 13 lines
arch/x86/kvm/svm/nested.c +6
-0@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ; +static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = {"static const", please.
Ugh, I was thinking the compiler would be magical enough to not generate code to fill an on-stack array at runtime, but that's not the case.
AFAICT, tagging it __initdata works, so I'll do this to hopefully ensure the memory is discarded after module load.
arch/x86/kvm/svm/nested.c +1 -1
@@ -205,7 +205,7 @@ static int svm_msrpm_offset(u32 msr)int __init nested_svm_init_msrpm_merge_offsets (void ) { -const u32 merge_msrs[] = { +static const u32 __initdata merge_msrs[] = {MSR_STAR ,MSR_IA32_SYSENTER_CS ,MSR_IA32_SYSENTER_EIP ,
Paolo Bonzini
maintainerRe: [PATCH 08/28] KVM: nSVM: Use dedicated array of MSRPM offsets to merge L0 and L1 bitmaps
In reply to Sean ChristophersonQuoted 27 lines · Sean Christopherson
Use a dedicated array of MSRPM offsets to merge L0 and L1 bitmaps, i.e. to merge KVM's vmcb01 bitmap with L1's vmcb12 bitmap. This will eventually allow for the removal of
direct_access_msrs, as the only path where tracking the offsets is truly justified is the merge for nested SVM, where merging in chunks is an easy way to batch uaccess reads/writes.Opportunistically omit the x2APIC MSRs from the merge-specific array instead of filtering them out at runtime.
Note, disabling interception of XSS, EFER, PAT, GHCB, and
TSC_AUXis mutually exclusive with nested virtualization, as KVM passes through the MSRs only for SEV-ES guests, and KVM doesn't support nested virtualization for SEV+ guests. Defer removing those MSRs to a future cleanup in order to make this refactoring as benign as possible.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/nested.c | 72 +++++++++++++++++++++++++++++++++------ arch/x86/kvm/svm/svm.c | 4 +++ arch/x86/kvm/svm/svm.h | 2 ++ 3 files changed, 67 insertions(+), 11 deletions(-)arch/x86/kvm/svm/nested.c +0
-0
arch/x86/kvm/svm/nested.c void recalc_intercepts(struct vcpu_svm *svm) · line 192
Quoted 9 lines
arch/x86/kvm/svm/nested.c +6
-0@@ -184,6 +184,64 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ; +static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; + +int __init nested_svm_init_msrpm_merge_offsets (void ) +{ +const u32 merge_msrs[] = {
"static const", please.
Paolo
Quoted 106 lines
arch/x86/kvm/svm/nested.c +55
-11+MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , + #ifdef CONFIG_X86_64 +MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , +MSR_CSTAR , +MSR_SYSCALL_MASK , + #endif +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , +MSR_IA32_FLUSH_CMD , +MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , + +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , + MSR_TSC_AUX, + }; +int i, j; + +for (i =0 ; i <ARRAY_SIZE (merge_msrs); i++) { +u32 offset =svm_msrpm_offset (merge_msrs[i]); + +if (WARN_ON (offset ==MSR_INVALID )) +return -EIO ; + +for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) { +if (nested_svm_msrpm_merge_offsets[j] == offset) +break ; + } + +if (j < nested_svm_nr_msrpm_merge_offsets) +continue ; + +if (WARN_ON (j >=ARRAY_SIZE (nested_svm_msrpm_merge_offsets))) +return -EIO ; + + nested_svm_msrpm_merge_offsets[j] = offset; + nested_svm_nr_msrpm_merge_offsets++; + } + +return 0 ; +} +/* * Merge L0's (KVM) and L1's (Nested VMCB) MSR permission bitmaps. The function * is optimized in that it only merges the parts where KVM MSR permission bitmap @@ -216,19 +274,11 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return true; -for (i =0 ; i <MSRPM_OFFSETS ; i++) { -u32 value, p; +for (i =0 ; i < nested_svm_nr_msrpm_merge_offsets; i++) { +const int p = nested_svm_msrpm_merge_offsets[i]; +u32 value;u64 offset; -if (msrpm_offsets[i] ==0xffffffff ) -break ; - - p = msrpm_offsets[i]; - -/* x2apic msrs are intercepted always for the nested guest */ -if (is_x2apic_msrpm_offset (p)) -continue ; - offset = svm->nested.ctl.msrpm_base_pa + (p *4 );if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))arch/x86/kvm/svm/svm.c +4
-0@@ -5689,6 +5689,10 @@ static int __init svm_init(void)if (!kvm_is_svm_supported ())return -EOPNOTSUPP ; + r =nested_svm_init_msrpm_merge_offsets (); +if (r) +return r; + r =kvm_x86_vendor_init (&svm_init_ops);if (r)return r;arch/x86/kvm/svm/svm.h +2
-0@@ -686,6 +686,8 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm *svm)return vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_NMI ); } +int __init nested_svm_init_msrpm_merge_offsets (void ); +int enter_svm_guest_mode (struct kvm_vcpu *vcpu,u64 vmcb_gpa,struct vmcb *vmcb12,bool from_vmrun);void svm_leave_nested (struct kvm_vcpu *vcpu);
Sean Christopherson
maintainerPATCH 9/28 KVM: nSVM: Omit SEV-ES specific passthrough MSRs from L0+L1 bitmap merge
In reply to Sean ChristophersonDon't merge bitmaps on nested VMRUN for MSRs that KVM passes through only for SEV-ES guests. KVM doesn't support nested virtualization for SEV-ES, and likely never will.
--- arch/x86/kvm/svm/nested.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/kvm/svm/nested.c +1 -7
@@ -184,7 +184,7 @@ void recalc_intercepts(struct vcpu_svm *svm) } } -static int nested_svm_msrpm_merge_offsets[9 ]__ro_after_init ; +static int nested_svm_msrpm_merge_offsets[6 ]__ro_after_init ;static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ;int __init nested_svm_init_msrpm_merge_offsets (void )
@@ -209,12 +209,6 @@ int __init nested_svm_init_msrpm_merge_offsets(void)MSR_IA32_LASTBRANCHTOIP ,MSR_IA32_LASTINTFROMIP , MSR_IA32_LASTINTTOIP, - -MSR_IA32_XSS , -MSR_EFER , -MSR_IA32_CR_PAT , -MSR_AMD64_SEV_ES_GHCB , - MSR_TSC_AUX, };int i, j;
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 10/28 KVM: nSVM: Don't initialize vmcb02 MSRPM with vmcb01's "always passthrough"
In reply to Sean ChristophersonDon't initialize vmcb02's MSRPM with KVM's set of "always passthrough" MSRs, as KVM always needs to consult L1's intercepts, i.e. needs to merge vmcb01 with vmcb12 and write the result to vmcb02. This will eventually allow for the removal of svm_vcpu_init_msrpm().
Note, the bitmaps are truly initialized by svm_vcpu_alloc_msrpm() (default to intercepting all MSRs), e.g. if there is a bug lurking elsewhere, the worst case scenario from dropping the call to svm_vcpu_init_msrpm() should be that KVM would fail to passthrough MSRs to L2.
--- arch/x86/kvm/svm/nested.c | 1 - arch/x86/kvm/svm/svm.c | 5 +++-- arch/x86/kvm/svm/svm.h | 1 - 3 files changed, 3 insertions(+), 4 deletions(-)
arch/x86/kvm/svm/nested.c +0 -1
@@ -1275,7 +1275,6 @@ int svm_allocate_nested(struct vcpu_svm *svm) svm->nested.msrpm =svm_vcpu_alloc_msrpm ();if (!svm->nested.msrpm) goto err_free_vmcb02; -svm_vcpu_init_msrpm (&svm->vcpu, svm->nested.msrpm); svm->nested.initialized = true;return 0 ;
arch/x86/kvm/svm/svm.c +3 -2
@@ -893,8 +893,9 @@ u32 *svm_vcpu_alloc_msrpm(void)return msrpm; } -void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu,u32 *msrpm) +static void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu) { +u32 *msrpm =to_svm (vcpu)->msrpm;int i;for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {
@@ -1403,7 +1404,7 @@ static void __svm_vcpu_reset(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); -svm_vcpu_init_msrpm (vcpu, svm->msrpm); +svm_vcpu_init_msrpm (vcpu);svm_init_osvw (vcpu);
arch/x86/kvm/svm/svm.h +0 -1
@@ -637,7 +637,6 @@ extern bool dump_invalid_vmcb;u32 svm_msrpm_offset (u32 msr);u32 *svm_vcpu_alloc_msrpm (void ); -void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu,u32 *msrpm);void svm_vcpu_free_msrpm (u32 *msrpm);void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb);void svm_enable_lbrv (struct kvm_vcpu *vcpu);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 11/28 KVM: SVM: Add helpers for accessing MSR bitmap that don't rely on offsets
In reply to Sean ChristophersonAdd macro-built helpers for testing, setting, and clearing MSRPM entries without relying on precomputed offsets. This sets the stage for eventually removing general KVM use of precomputed offsets, which are quite confusing and rather inefficient for the vast majority of KVM's usage.
Outside of merging L0 and L1 bitmaps for nested SVM, using u32-indexed offsets and accesses is at best unnecessary, and at worst introduces extra operations to retrieve the individual bit from within the offset u32 value. And simply calling them "offsets" is very confusing, as the "unit" of the offset isn't immediately obvious.
Use the new helpers in set_msr_interception_bitmap() and msr_write_intercepted() to verify the math and operations, but keep the existing offset-based logic set_msr_interception_bitmap() to sanity check the "clear" and "set" operations. Manipulating MSR interceptions isn't a hot path and no kernel release is ever expected to contain this specific version of set_msr_interception_bitmap() (it will be removed entirely in the near future).
Add compile-time asserts to verify the bit number calculations, and also to provide a simple demonstration of the layout (SVM and VMX use the same concept of a bitmap, but with different layouts).
--- arch/x86/kvm/svm/svm.c | 30 ++++++++++++++-------------- arch/x86/kvm/svm/svm.h | 44 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/svm.c +14 -16
@@ -806,11 +806,6 @@ static bool valid_msr_intercept(u32 index)static bool msr_write_intercepted (struct kvm_vcpu *vcpu,u32 msr) { -u8 bit_write; -unsigned long tmp; -u32 offset; -u32 *msrpm; -/* * For non-nested case: * If the L01 MSR bitmap does not intercept the MSR, then we need to
@@ -820,17 +815,10 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr)* If the L02 MSR bitmap does not intercept the MSR, then we need to * save it. */ - msrpm =is_guest_mode (vcpu) ?to_svm (vcpu)->nested.msrpm: -to_svm (vcpu)->msrpm; +void *msrpm =is_guest_mode (vcpu) ?to_svm (vcpu)->nested.msrpm: +to_svm (vcpu)->msrpm; - offset =svm_msrpm_offset (msr); - bit_write =2 * (msr &0x0f ) +1 ; - tmp = msrpm[offset]; - -if (KVM_BUG_ON (offset ==MSR_INVALID , vcpu->kvm)) -return false; - -return test_bit (bit_write, &tmp); +return svm_test_msr_bitmap_write (msrpm, msr); } static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm,
@@ -865,7 +853,17 @@ static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm, read ?__clear_bit (bit_read, &tmp) :__set_bit (bit_read, &tmp); write ?__clear_bit (bit_write, &tmp) :__set_bit (bit_write, &tmp); - msrpm[offset] = tmp; +if (read) +svm_clear_msr_bitmap_read ((void *)msrpm, msr); +else +svm_set_msr_bitmap_read ((void *)msrpm, msr); + +if (write) +svm_clear_msr_bitmap_write ((void *)msrpm, msr); +else +svm_set_msr_bitmap_write ((void *)msrpm, msr); + +WARN_ON_ONCE (msrpm[offset] != (u32 )tmp);svm_hv_vmcb_dirty_nested_enlightenments (vcpu); svm->nested.force_msr_bitmap_recalc = true;
arch/x86/kvm/svm/svm.h +44 -0
50 lines not on this page · raw message
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Paolo Bonzini
maintainerRe: [PATCH 11/28] KVM: SVM: Add helpers for accessing MSR bitmap that don't rely on offsets
Quoted 16 lines · Sean Christopherson
arch/x86/kvm/svm/svm.h +8
-0@@ -628,6 +628,50 @@ static_assert(SVM_MSRS_PER_RANGE == 8192);#define SVM_MSRPM_RANGE_1_BASE_MSR 0xc0000000#define SVM_MSRPM_RANGE_2_BASE_MSR 0xc0010000 +#define SVM_MSRPM_FIRST_MSR (range_nr) \ + (SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) +#define SVM_MSRPM_LAST_MSR (range_nr) \ + (SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR + SVM_MSRS_PER_RANGE - 1) + +#define SVM_MSRPM_BIT_NR (range_nr, msr) \ + (range_nr * SVM_MSRPM_BYTES_PER_RANGE * BITS_PER_BYTE + \ + (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) * SVM_BITS_PER_MSR)
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 648
Quoted 10 lines
arch/x86/kvm/svm/svm.h +10
-0+ +#define SVM_MSRPM_SANITY_CHECK_BITS (range_nr) \ +static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ + range_nr * 2048 * 8 + 2); \ +static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ + range_nr * 2048 * 8 + 14); + +SVM_MSRPM_SANITY_CHECK_BITS (0 ); +SVM_MSRPM_SANITY_CHECK_BITS (1 ); +SVM_MSRPM_SANITY_CHECK_BITS (2 );
Replying here for patches 11/25/26. None of this is needed, just write a function like this:
static inline u32 svm_msr_bit(u32 msr)
{
u32 msr_base = msr & ~(SVM_MSRS_PER_RANGE - 1);
if (msr_base == SVM_MSRPM_RANGE_0_BASE_MSR)
return SVM_MSRPM_BIT_NR(0, msr);
if (msr_base == SVM_MSRPM_RANGE_1_BASE_MSR)
return SVM_MSRPM_BIT_NR(1, msr);
if (msr_base == SVM_MSRPM_RANGE_2_BASE_MSR)
return SVM_MSRPM_BIT_NR(2, msr);
return MSR_INVALID;
}and you can throw away most of the other macros. For example:
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 652
Quoted 3 lines
arch/x86/kvm/svm/svm.h +3
-0+#define SVM_BUILD_MSR_BITMAP_CASE (bitmap, range_nr, msr, bitop, bit_rw) \ + case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \ + return bitop##_bit(SVM_MSRPM_BIT_NR(range_nr, msr) + bit_rw, bitmap);
... becomes a lot more lowercase:
static inline rtype svm_##action##_msr_bitmap_##access(
unsigned long *bitmap, u32 msr)
{
u32 bit = svm_msr_bit(msr);
if (bit == MSR_INVALID)
return true;
return bitop##_bit(bit + bit_rw, bitmap);
}In patch 25, also, you just get
static u32 svm_msrpm_offset(u32 msr)
{
u32 bit = svm_msr_bit(msr);
if (bit == MSR_INVALID)
return MSR_INVALID;
return bit / BITS_PER_BYTE;
}And you change everything to -EINVAL in patch 26 to kill MSR_INVALID.
Another nit...
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 673
Quoted 7 lines
arch/x86/kvm/svm/svm.h +7
-0+#define BUILD_SVM_MSR_BITMAP_HELPERS (ret_type, action, bitop) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 1) + +BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test) +BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear) +BUILD_SVM_MSR_BITMAP_HELPERS (void, set, __set)
Yes it's a bit duplication, but no need for the nesting, just do:
BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test, write, 1) BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear, write, 1) BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set, write, 1)
Otherwise, really nice.
Paolo
Sean Christopherson
maintainerRe: [PATCH 11/28] KVM: SVM: Add helpers for accessing MSR bitmap that don't rely on offsets
Quoted 30 lines · Paolo Bonzini
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/svm/svm.h +8
-0@@ -628,6 +628,50 @@ static_assert(SVM_MSRS_PER_RANGE == 8192);#define SVM_MSRPM_RANGE_1_BASE_MSR 0xc0000000#define SVM_MSRPM_RANGE_2_BASE_MSR 0xc0010000 +#define SVM_MSRPM_FIRST_MSR (range_nr) \ + (SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) +#define SVM_MSRPM_LAST_MSR (range_nr) \ + (SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR + SVM_MSRS_PER_RANGE - 1) + +#define SVM_MSRPM_BIT_NR (range_nr, msr) \ + (range_nr * SVM_MSRPM_BYTES_PER_RANGE * BITS_PER_BYTE + \ + (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) * SVM_BITS_PER_MSR)arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 647
Quoted 10 lines
arch/x86/kvm/svm/svm.h +10
-0+ +#define SVM_MSRPM_SANITY_CHECK_BITS (range_nr) \ +static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ + range_nr * 2048 * 8 + 2); \ +static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ + range_nr * 2048 * 8 + 14); + +SVM_MSRPM_SANITY_CHECK_BITS (0 ); +SVM_MSRPM_SANITY_CHECK_BITS (1 ); +SVM_MSRPM_SANITY_CHECK_BITS (2 );Replying here for patches 11/25/26. None of this is needed, just write a function like this:
Quoted 3 lines
static inline u32
svm_msr_bit(u32 msr){ u32 msr_base = msr & ~(SVM_MSRS_PER_RANGE - 1);
Ooh, clever.
Quoted 7 lines
if (msr_base == SVM_MSRPM_RANGE_0_BASE_MSR) return SVM_MSRPM_BIT_NR(0, msr); if (msr_base == SVM_MSRPM_RANGE_1_BASE_MSR) return SVM_MSRPM_BIT_NR(1, msr); if (msr_base == SVM_MSRPM_RANGE_2_BASE_MSR) return SVM_MSRPM_BIT_NR(2, msr); return MSR_INVALID;
I initially had something like this, but I don't like the potential for typos, e.g. to fat finger something like:
if (msr_base == SVM_MSRPM_RANGE_2_BASE_MSR) return SVM_MSRPM_BIT_NR(1, msr);
Which is how I ended up with the (admittedly ugly) CASE macros. Would you be ok keeping that wrinkle? E.g.
#define SVM_MSR_BIT_NR_CASE(range_nr, msr) \
case SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR: \
return range_nr * SVM_MSRPM_BYTES_PER_RANGE * BITS_PER_BYTE + \
(msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) * SVM_BITS_PER_MSR);
static __always_inline int svm_msrpm_bit_nr(u32 msr)
{
switch (msr & ~(SVM_MSRS_PER_RANGE - 1)) {
SVM_BUILD_MSR_BITMAP_CASE(0, msr)
SVM_BUILD_MSR_BITMAP_CASE(1, msr)
SVM_BUILD_MSR_BITMAP_CASE(2, msr)
default:
return -EINVAL;
}
}Actually, better idea! Hopefully. With your masking trick, there's no need to do subtraction to get the offset within a range, which means getting the bit/byte number for an MSR can be done entirely programmatically. And if we do that, then the SVM_MSRPM_RANGE_xxx_BASE_MSR defines can go away, and the (very trivial) copy+paste that I dislike also goes away.
Completely untested, but how about this?
#define SVM_MSRPM_OFFSET_MASK (SVM_MSRS_PER_RANGE - 1)
static __always_inline int svm_msrpm_bit_nr(u32 msr)
{
int range_nr;
switch (msr & ~SVM_MSRPM_OFFSET_MASK) {
case 0:
range_nr = 0;
break;
case 0xc0000000:
range_nr = 1;
break;
case 0xc0010000:
range_nr = 2;
break;
default:
return -EINVAL;
}
return range_nr * SVM_MSRPM_BYTES_PER_RANGE * BITS_PER_BYTE +
(msr & SVM_MSRPM_OFFSET_MASK) * SVM_BITS_PER_MSR)
}
static inline svm_msrpm_byte_nr(u32 msr)
{
return svm_msrpm_bit_nr(msr) / BITS_PER_BYTE;
}The open coded literals aren't pretty, but VMX does the same thing, precisely because I didn't want any code besides the innermost helper dealing with the msr => offset math.
Quoted 10 lines
}
and you can throw away most of the other macros. For example:
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 652
Quoted 3 lines
arch/x86/kvm/svm/svm.h +3
-0+#define SVM_BUILD_MSR_BITMAP_CASE (bitmap, range_nr, msr, bitop, bit_rw) \ + case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \ + return bitop##_bit(SVM_MSRPM_BIT_NR(range_nr, msr) + bit_rw, bitmap);... becomes a lot more lowercase:
Quoted 7 lines
static inline rtype svm_##action##_msr_bitmap_##access(
unsigned long *bitmap, u32 msr){ u32 bit = svm_msr_bit(msr); if (bit == MSR_INVALID) return true; return bitop##_bit(bit + bit_rw, bitmap);
Yeah, much cleaner.
Quoted 17 lines
}
In patch 25, also, you just get
static u32
svm_msrpm_offset(u32 msr){ u32 bit = svm_msr_bit(msr); if (bit == MSR_INVALID) return MSR_INVALID; return bit / BITS_PER_BYTE; }And you change everything to
-EINVALin patch 26 to killMSR_INVALID.Another nit...
Quoted 8 lines
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 673
Quoted 7 lines
arch/x86/kvm/svm/svm.h +7
-0+#define BUILD_SVM_MSR_BITMAP_HELPERS (ret_type, action, bitop) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 1) + +BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test) +BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear) +BUILD_SVM_MSR_BITMAP_HELPERS (void, set, __set)Yes it's a bit duplication, but no need for the nesting, just do:
I don't have a super strong preference, but I do want to be consistent between VMX and SVM, and VMX has the nesting (unsurprisingly, also written by me). And for that, the nested macros add a bit more value due to reads vs writes being in entirely different areas of the bitmap.
#define BUILD_VMX_MSR_BITMAP_HELPERS(ret_type, action, bitop) \ __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0x0) \ __BUILD_VMX_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 0x800)
BUILD_VMX_MSR_BITMAP_HELPERS(bool, test, test) BUILD_VMX_MSR_BITMAP_HELPERS(void, clear, __clear) BUILD_VMX_MSR_BITMAP_HELPERS(void, set, __set)
That could be mitigated to some extent by adding a #define to communicate the offset, but IMO the nested macros are less ugly than that.
Quoted 11 lines
BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test, write, 1) BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear, write, 1) BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set, read, 0) BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set, write, 1)Otherwise, really nice.
Paolo
Paolo Bonzini
maintainerRe: [PATCH 11/28] KVM: SVM: Add helpers for accessing MSR bitmap that don't rely on offsets
Quoted 21 lines · Sean Christopherson
Quoted 6 lines · Paolo Bonzini
Replying here for patches 11/25/26. None of this is needed, just write a function like this:
static inline u32
svm_msr_bit(u32 msr){ u32 msr_base = msr & ~(SVM_MSRS_PER_RANGE - 1);Ooh, clever.
Quoted 7 lines
if (msr_base == SVM_MSRPM_RANGE_0_BASE_MSR) return SVM_MSRPM_BIT_NR(0, msr); if (msr_base == SVM_MSRPM_RANGE_1_BASE_MSR) return SVM_MSRPM_BIT_NR(1, msr); if (msr_base == SVM_MSRPM_RANGE_2_BASE_MSR) return SVM_MSRPM_BIT_NR(2, msr); return MSR_INVALID;I initially had something like this, but I don't like the potential for typos, e.g. to fat finger something like:
Quoted 7 lines
if (msr_base == SVM_MSRPM_RANGE_2_BASE_MSR) return SVM_MSRPM_BIT_NR(1, msr);Which is how I ended up with the (admittedly ugly) CASE macros. [...]
Actually, better idea! Hopefully. With your masking trick, there's no need to
do subtraction to get the offset within a range, which means getting the bit/byte
number for an MSR can be done entirely programmatically. And if we do that, then> theSVM_MSRPM_RANGE_xxx_BASE_MSRdefines can go away, and the (very
trivial)
Quoted 7 lines
copy+paste that I dislike also goes away.
Completely untested, but how about this?
#define SVM_MSRPM_OFFSET_MASK (SVM_MSRS_PER_RANGE - 1) static __always_inline int svm_msrpm_bit_nr(u32 msr)
(yeah, after hitting send I noticed that msr->msrpm would have been better)
Quoted 8 lines
{
int range_nr;
switch (msr & ~SVM_MSRPM_OFFSET_MASK) {
case 0:
range_nr = 0;
break;
case 0xc0000000:Quoted 8 lines
range_nr = 1; break; case 0xc0010000: range_nr = 2; break; default: return -EINVAL; }
I actually was going to propose something very similar, I refrained only because I wasn't sure if there would be other remaining uses of SVM_MSRPM_RANGE_?_BASE_MSR. The above is nice.
Quoted 2 lines
return
range_nr*SVM_MSRPM_BYTES_PER_RANGE*BITS_PER_BYTE+
(msr &SVM_MSRPM_OFFSET_MASK) *SVM_BITS_PER_MSR)
Or this too:
return ((range_nr * SVM_MSRS_PER_RANGE)
+ (msr & SVM_MSRPM_OFFSET_MASK)) * SVM_BITS_PER_MSR;
depending on personal taste. A few less macros, a few more parentheses.
That removes the enjoyment of seeing everything collapse into a single LEA instruction (X*2+CONST), as was the case with SVM_MSRPM_BIT_NR. But I agree that these versions are about as nice as the code can be made.
Quoted 13 lines
The open coded literals aren't pretty, but VMX does the same thing, precisely because I didn't want any code besides the innermost helper dealing with the msr => offset math.
Quoted 8 lines
arch/x86/kvm/svm/svm.h static_assert(SVM_MSRS_PER_RANGE == 8192); · line 673
Quoted 7 lines
arch/x86/kvm/svm/svm.h +7
-0+#define BUILD_SVM_MSR_BITMAP_HELPERS (ret_type, action, bitop) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, read, 0) \ + __BUILD_SVM_MSR_BITMAP_HELPER(ret_type, action, bitop, write, 1) + +BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test) +BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear) +BUILD_SVM_MSR_BITMAP_HELPERS (void, set, __set)Yes it's a bit duplication, but no need for the nesting, just do:
Quoted 4 lines
I don't have a super strong preference, but I do want to be consistent between VMX and SVM, and VMX has the nesting (unsurprisingly, also written by me). And for that, the nested macros add a bit more value due to reads vs writes being in entirely different areas of the bitmap.
Yeah, fair enough. Since it's copied from VMX it makes sense.
Paolo
Sean Christopherson
maintainerPATCH 12/28 KVM: SVM: Implement and adopt VMX style MSR intercepts APIs
In reply to Sean ChristophersonAdd and use SVM MSR interception APIs (in most paths) to match VMX's APIs and nomenclature. Specifically, add SVM variants of:
vmx_disable_intercept_for_msr(vcpu, msr, type) vmx_enable_intercept_for_msr(vcpu, msr, type) vmx_set_intercept_for_msr(vcpu, msr, type, intercept)
to eventually replace SVM's single helper:
set_msr_interception(vcpu, msrpm, msr, allow_read, allow_write)
which is awkward to use (in all cases, KVM either applies the same logic for both reads and writes, or intercepts one of read or write), and is unintuitive due to using '0' to indicate interception should be *set*.
Keep the guts of the old API for the moment to avoid churning the MSR filter code, as that mess will be overhauled in the near future. Leave behind a temporary comment to call out that the shadow bitmaps have inverted polarity relative to the bitmaps consumed by hardware.
No functional change intended.
--- arch/x86/kvm/svm/sev.c | 18 ++++---- arch/x86/kvm/svm/svm.c | 100 ++++++++++++++++++++++++++++++----------- arch/x86/kvm/svm/svm.h | 12 +++++ 3 files changed, 93 insertions(+), 37 deletions(-)
arch/x86/kvm/svm/sev.c +8 -10
@@ -4415,12 +4415,10 @@ static void sev_es_vcpu_after_set_cpuid(struct vcpu_svm *svm) {struct kvm_vcpu *vcpu = &svm->vcpu; -if (boot_cpu_has (X86_FEATURE_V_TSC_AUX )) { -bool v_tsc_aux =guest_cpu_cap_has (vcpu,X86_FEATURE_RDTSCP ) || -guest_cpu_cap_has (vcpu,X86_FEATURE_RDPID ); - -set_msr_interception (vcpu, svm->msrpm,MSR_TSC_AUX , v_tsc_aux, v_tsc_aux); - } +if (boot_cpu_has (X86_FEATURE_V_TSC_AUX )) +svm_set_intercept_for_msr (vcpu,MSR_TSC_AUX ,MSR_TYPE_RW , + !guest_cpu_cap_has (vcpu,X86_FEATURE_RDTSCP ) && + !guest_cpu_cap_has (vcpu,X86_FEATURE_RDPID ));/* * For SEV-ES, accesses to MSR_IA32_XSS should not be intercepted if
@@ -4436,9 +4434,9 @@ static void sev_es_vcpu_after_set_cpuid(struct vcpu_svm *svm)*/ if (guest_cpu_cap_has (vcpu,X86_FEATURE_XSAVES ) &&guest_cpuid_has (vcpu,X86_FEATURE_XSAVES )) -set_msr_interception (vcpu, svm->msrpm,MSR_IA32_XSS ,1 ,1 ); +svm_disable_intercept_for_msr (vcpu,MSR_IA32_XSS ,MSR_TYPE_RW );else -set_msr_interception (vcpu, svm->msrpm,MSR_IA32_XSS ,0 ,0 ); +svm_enable_intercept_for_msr (vcpu,MSR_IA32_XSS ,MSR_TYPE_RW ); }void sev_vcpu_after_set_cpuid (struct vcpu_svm *svm)
@@ -4515,8 +4513,8 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm)svm_clr_intercept (svm,INTERCEPT_XSETBV );/* Clear intercepts on selected MSRs */ -set_msr_interception (vcpu, svm->msrpm,MSR_EFER ,1 ,1 ); -set_msr_interception (vcpu, svm->msrpm,MSR_IA32_CR_PAT ,1 ,1 ); +svm_disable_intercept_for_msr (vcpu,MSR_EFER ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_IA32_CR_PAT ,MSR_TYPE_RW ); }void sev_init_vmcb (struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.c +73 -27
170 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +12 -0
@@ -701,6 +701,18 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool disable);void svm_complete_interrupt_delivery (struct kvm_vcpu *vcpu,int delivery_mode,int trig_mode,int vec); +void svm_enable_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type); +void svm_disable_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type); + +static inline void svm_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr, +int type,bool enable_intercept) +{ +if (enable_intercept) +svm_enable_intercept_for_msr (vcpu, msr, type); +else +svm_disable_intercept_for_msr (vcpu, msr, type); +} +/* nested.c */ #define NESTED_EXIT_HOST 0/* Exit handled on host level */
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 12/28] KVM: SVM: Implement and adopt VMX style MSR intercepts APIs
Quoted 15 lines · Sean Christopherson
Add and use SVM MSR interception APIs (in most paths) to match VMX's APIs and nomenclature. Specifically, add SVM variants of:
vmx_disable_intercept_for_msr(vcpu, msr, type)vmx_enable_intercept_for_msr(vcpu, msr, type)vmx_set_intercept_for_msr(vcpu, msr, type, intercept)to eventually replace SVM's single helper:
set_msr_interception(vcpu, msrpm, msr, allow_read, allow_write)which is awkward to use (in all cases, KVM either applies the same logic for both reads and writes, or intercepts one of read or write), and is unintuitive due to using '0' to indicate interception should be *set*.
Quoted 8 lines
Keep the guts of the old API for the moment to avoid churning the MSR filter code, as that mess will be overhauled in the near future. Leave behind a temporary comment to call out that the shadow bitmaps have inverted polarity relative to the bitmaps consumed by hardware.
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
two nits below:
arch/x86/kvm/svm/svm.c static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm, · line 881
Quoted 12 lines
+void
svm_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type){arch/x86/kvm/svm/svm.c +8
-2-set_shadow_msr_intercept (vcpu, msr, read, write); -set_msr_interception_bitmap (vcpu, msrpm, msr, read, write); +struct vcpu_svm *svm =to_svm (vcpu); +void *msrpm = svm->msrpm; + +/* Note, the shadow intercept bitmaps have inverted polarity. */ +set_shadow_msr_intercept (vcpu, msr, type &MSR_TYPE_R , type &MSR_TYPE_W ); + +/* +* Don't disabled interception for the MSR if userspace wants to
s/disabled/disable
<snip>
arch/x86/kvm/svm/svm.c static void set_msr_interception_bitmap(struct kvm_vcpu *vcpu, u32 *msrpm, · line 911
Quoted 6 lines
arch/x86/kvm/svm/svm.c +6
-0+void svm_enable_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type) +{ +struct vcpu_svm *svm =to_svm (vcpu); +void *msrpm = svm->msrpm; + +
Remove one newline here.
Sean Christopherson
maintainerPATCH 13/28 KVM: SVM: Pass through GHCB MSR if and only if VM is an SEV-ES guest
In reply to Sean ChristophersonDisable interception of the GHCB MSR if and only if the VM is an SEV-ES guest. While the exact behavior is completely undocumented in the APM, common sense and testing on SEV-ES capable CPUs says that accesses to the GHCB from non-SEV-ES guests will #GP. I.e. from the guest's perspective, no functional change intended.
--- arch/x86/kvm/svm/sev.c | 3 ++- arch/x86/kvm/svm/svm.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/kvm/svm/sev.c +2 -1
@@ -4512,7 +4512,8 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm)/* Can't intercept XSETBV, HV can't modify XCR0 directly */ svm_clr_intercept (svm,INTERCEPT_XSETBV ); -/* Clear intercepts on selected MSRs */ +/* Clear intercepts on MSRs that are context switched by hardware. */ +svm_disable_intercept_for_msr (vcpu,MSR_AMD64_SEV_ES_GHCB ,MSR_TYPE_RW );svm_disable_intercept_for_msr (vcpu,MSR_EFER ,MSR_TYPE_RW );svm_disable_intercept_for_msr (vcpu,MSR_IA32_CR_PAT ,MSR_TYPE_RW ); }
arch/x86/kvm/svm/svm.c +1 -1
@@ -110,7 +110,7 @@ static const struct svm_direct_access_msrs { { .index = MSR_IA32_XSS, .always = false }, { .index = MSR_EFER, .always = false }, { .index = MSR_IA32_CR_PAT, .always = false }, - { .index = MSR_AMD64_SEV_ES_GHCB, .always = true }, + { .index = MSR_AMD64_SEV_ES_GHCB, .always = false }, { .index = MSR_TSC_AUX, .always = false }, { .index = X2APIC_MSR(APIC_ID), .always = false }, { .index = X2APIC_MSR(APIC_LVR), .always = false },
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 14/28 KVM: SVM: Drop "always" flag from list of possible passthrough MSRs
In reply to Sean ChristophersonDrop the "always" flag from the array of possible passthrough MSRs, and instead manually initialize the permissions for the handful of MSRs that KVM passes through by default. In addition to cutting down on boilerplate copy+paste code and eliminating a misleading flag (the MSRs aren't always passed through, e.g. thanks to MSR filters), this will allow for removing the direct_access_msrs array entirely.
--- arch/x86/kvm/svm/svm.c | 124 ++++++++++++++++++++--------------------- 1 file changed, 62 insertions(+), 62 deletions(-)
arch/x86/kvm/svm/svm.c +62 -62
@@ -83,51 +83,48 @@ static DEFINE_PER_CPU(u64, current_tsc_ratio);#define X2APIC_MSR (x) (APIC_BASE_MSR + (x >> 4)) -static const struct svm_direct_access_msrs { -u32 index;/* Index of the MSR */ -bool always;/* True if intercept is initially cleared */ -} direct_access_msrs[] = { - { .index =MSR_STAR , .always = true }, - { .index =MSR_IA32_SYSENTER_CS , .always = true }, - { .index =MSR_IA32_SYSENTER_EIP , .always = false }, - { .index =MSR_IA32_SYSENTER_ESP , .always = false }, +static const u32 direct_access_msrs[] = { +MSR_STAR , +MSR_IA32_SYSENTER_CS , +MSR_IA32_SYSENTER_EIP , +MSR_IA32_SYSENTER_ESP , #ifdef CONFIG_X86_64 - { .index =MSR_GS_BASE , .always = true }, - { .index =MSR_FS_BASE , .always = true }, - { .index =MSR_KERNEL_GS_BASE , .always = true }, - { .index =MSR_LSTAR , .always = true }, - { .index =MSR_CSTAR , .always = true }, - { .index =MSR_SYSCALL_MASK , .always = true }, +MSR_GS_BASE , +MSR_FS_BASE , +MSR_KERNEL_GS_BASE , +MSR_LSTAR , +MSR_CSTAR , +MSR_SYSCALL_MASK , #endif - { .index =MSR_IA32_SPEC_CTRL , .always = false }, - { .index =MSR_IA32_PRED_CMD , .always = false }, - { .index =MSR_IA32_FLUSH_CMD , .always = false }, - { .index =MSR_IA32_DEBUGCTLMSR , .always = false }, - { .index =MSR_IA32_LASTBRANCHFROMIP , .always = false }, - { .index =MSR_IA32_LASTBRANCHTOIP , .always = false }, - { .index =MSR_IA32_LASTINTFROMIP , .always = false }, - { .index =MSR_IA32_LASTINTTOIP , .always = false }, - { .index =MSR_IA32_XSS , .always = false }, - { .index =MSR_EFER , .always = false }, - { .index =MSR_IA32_CR_PAT , .always = false }, - { .index =MSR_AMD64_SEV_ES_GHCB , .always = false }, - { .index =MSR_TSC_AUX , .always = false }, - { .index =X2APIC_MSR (APIC_ID ), .always = false }, - { .index =X2APIC_MSR (APIC_LVR ), .always = false }, - { .index =X2APIC_MSR (APIC_TASKPRI ), .always = false }, - { .index =X2APIC_MSR (APIC_ARBPRI ), .always = false }, - { .index =X2APIC_MSR (APIC_PROCPRI ), .always = false }, - { .index =X2APIC_MSR (APIC_EOI ), .always = false }, - { .index =X2APIC_MSR (APIC_RRR ), .always = false }, - { .index =X2APIC_MSR (APIC_LDR ), .always = false }, - { .index =X2APIC_MSR (APIC_DFR ), .always = false }, - { .index =X2APIC_MSR (APIC_SPIV ), .always = false }, - { .index =X2APIC_MSR (APIC_ISR ), .always = false }, - { .index =X2APIC_MSR (APIC_TMR ), .always = false }, - { .index =X2APIC_MSR (APIC_IRR ), .always = false }, - { .index =X2APIC_MSR (APIC_ESR ), .always = false }, - { .index =X2APIC_MSR (APIC_ICR ), .always = false }, - { .index =X2APIC_MSR (APIC_ICR2 ), .always = false }, +MSR_IA32_SPEC_CTRL , +MSR_IA32_PRED_CMD , +MSR_IA32_FLUSH_CMD , +MSR_IA32_DEBUGCTLMSR , +MSR_IA32_LASTBRANCHFROMIP , +MSR_IA32_LASTBRANCHTOIP , +MSR_IA32_LASTINTFROMIP , +MSR_IA32_LASTINTTOIP , +MSR_IA32_XSS , +MSR_EFER , +MSR_IA32_CR_PAT , +MSR_AMD64_SEV_ES_GHCB , +MSR_TSC_AUX , +X2APIC_MSR (APIC_ID ), +X2APIC_MSR (APIC_LVR ), +X2APIC_MSR (APIC_TASKPRI ), +X2APIC_MSR (APIC_ARBPRI ), +X2APIC_MSR (APIC_PROCPRI ), +X2APIC_MSR (APIC_EOI ), +X2APIC_MSR (APIC_RRR ), +X2APIC_MSR (APIC_LDR ), +X2APIC_MSR (APIC_DFR ), +X2APIC_MSR (APIC_SPIV ), +X2APIC_MSR (APIC_ISR ), +X2APIC_MSR (APIC_TMR ), +X2APIC_MSR (APIC_IRR ), +X2APIC_MSR (APIC_ESR ), +X2APIC_MSR (APIC_ICR ), +X2APIC_MSR (APIC_ICR2 ),/* * Note:
@@ -136,14 +133,14 @@ static const struct svm_direct_access_msrs {* the AVIC hardware would generate GP fault. Therefore, always * intercept the MSR 0x832, and do not setup direct_access_msr. */ - { .index = X2APIC_MSR(APIC_LVTTHMR), .always = false }, - { .index = X2APIC_MSR(APIC_LVTPC), .always = false }, - { .index = X2APIC_MSR(APIC_LVT0), .always = false }, - { .index = X2APIC_MSR(APIC_LVT1), .always = false }, - { .index = X2APIC_MSR(APIC_LVTERR), .always = false }, - { .index = X2APIC_MSR(APIC_TMICT), .always = false }, - { .index = X2APIC_MSR(APIC_TMCCT), .always = false }, - { .index = X2APIC_MSR(APIC_TDCR), .always = false }, +X2APIC_MSR (APIC_LVTTHMR ), +X2APIC_MSR (APIC_LVTPC ), +X2APIC_MSR (APIC_LVT0 ), +X2APIC_MSR (APIC_LVT1 ), +X2APIC_MSR (APIC_LVTERR ), +X2APIC_MSR (APIC_TMICT ), +X2APIC_MSR (APIC_TMCCT ), +X2APIC_MSR (APIC_TDCR ), }; static_assert(ARRAY_SIZE(direct_access_msrs) ==
@@ -771,7 +768,7 @@ static int direct_access_msr_slot(u32 msr)u32 i;for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) { -if (direct_access_msrs[i].index == msr) +if (direct_access_msrs[i] == msr)return i; }
@@ -939,14 +936,17 @@ u32 *svm_vcpu_alloc_msrpm(void)static void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu) { -int i; +svm_disable_intercept_for_msr (vcpu,MSR_STAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); -for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) { -if (!direct_access_msrs[i].always) -continue ; -svm_disable_intercept_for_msr (vcpu, direct_access_msrs[i].index, -MSR_TYPE_RW ); - } +#ifdef CONFIG_X86_64 +svm_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_LSTAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_CSTAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_SYSCALL_MASK ,MSR_TYPE_RW ); +#endif }void svm_set_x2apic_msr_interception (struct vcpu_svm *svm,bool intercept)
@@ -960,7 +960,7 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept)return ;for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) { -int index = direct_access_msrs[i].index; +int index = direct_access_msrs[i];if ((index <APIC_BASE_MSR ) || (index >APIC_BASE_MSR +0xff ))
@@ -988,7 +988,7 @@ static void svm_msr_filter_changed(struct kvm_vcpu *vcpu)* back in sync after this. */ for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) { -u32 msr = direct_access_msrs[i].index; +u32 msr = direct_access_msrs[i];u32 read =test_bit (i, svm->shadow_msr_intercept.read);u32 write =test_bit (i, svm->shadow_msr_intercept.write);
@@ -1028,7 +1028,7 @@ static __init int init_msrpm_offsets(void)for (i =0 ; i <ARRAY_SIZE (direct_access_msrs); i++) {u32 offset; - offset =svm_msrpm_offset (direct_access_msrs[i].index); + offset =svm_msrpm_offset (direct_access_msrs[i]);if (WARN_ON (offset ==MSR_INVALID ))return -EIO ;
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 15/28 KVM: x86: Move definition of X2APIC_MSR() to lapic.h
In reply to Sean ChristophersonDedup the definition of X2APIC_MSR and put it in the local APIC code where it belongs.
No functional change intended.
--- arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/svm/svm.c | 2 -- arch/x86/kvm/vmx/vmx.h | 2 -- 3 files changed, 2 insertions(+), 4 deletions(-)
arch/x86/kvm/lapic.h +2 -0
@@ -21,6 +21,8 @@#define APIC_BROADCAST 0xFF#define X2APIC_BROADCAST 0xFFFFFFFFul +#define X2APIC_MSR (r) (APIC_BASE_MSR + ((r) >> 4)) +enum lapic_mode {LAPIC_MODE_DISABLED =0 ,LAPIC_MODE_INVALID =X2APIC_ENABLE ,
arch/x86/kvm/svm/svm.c +0 -2
@@ -81,8 +81,6 @@ static uint64_t osvw_len = 4, osvw_status; static DEFINE_PER_CPU(u64, current_tsc_ratio); -#define X2APIC_MSR (x) (APIC_BASE_MSR + (x >> 4)) -static const u32 direct_access_msrs[] = {MSR_STAR ,MSR_IA32_SYSENTER_CS ,
arch/x86/kvm/vmx/vmx.h +0 -2
@@ -19,8 +19,6 @@#include "../mmu.h" #include "common.h" -#define X2APIC_MSR (r) (APIC_BASE_MSR + ((r) >> 4)) -#ifdef CONFIG_X86_64 #define MAX_NR_USER_RETURN_MSRS 7#else
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 16/28 KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter change
In reply to Sean ChristophersonOn a userspace MSR filter change, recalculate all MSR intercepts using the filter-agnostic logic instead of maintaining a "shadow copy" of KVM's desired intercepts. The shadow bitmaps add yet another point of failure, are confusing (e.g. what does "handled specially" mean!?!?), an eyesore, and a maintenance burden.
Given that KVM *must* be able to recalculate the correct intercepts at any given time, and that MSR filter updates are not hot paths, there is zero benefit to maintaining the shadow bitmaps.
--- arch/x86/kvm/vmx/vmx.c | 184 +++++++++++------------------------------ arch/x86/kvm/vmx/vmx.h | 7 -- 2 files changed, 47 insertions(+), 144 deletions(-)
arch/x86/kvm/vmx/vmx.c +47 -137
@@ -166,31 +166,6 @@ module_param(allow_smaller_maxphyaddr, bool, S_IRUGO);RTIT_STATUS_ERROR |RTIT_STATUS_STOPPED | \ RTIT_STATUS_BYTECNT)) -/* -* List of MSRs that can be directly passed to the guest. -* In addition to these x2apic, PT and LBR MSRs are handled specially. -*/ -static u32 vmx_possible_passthrough_msrs[MAX_POSSIBLE_PASSTHROUGH_MSRS ] = { -MSR_IA32_SPEC_CTRL , -MSR_IA32_PRED_CMD , -MSR_IA32_FLUSH_CMD , -MSR_IA32_TSC , -#ifdef CONFIG_X86_64 -MSR_FS_BASE , -MSR_GS_BASE , -MSR_KERNEL_GS_BASE , -MSR_IA32_XFD , -MSR_IA32_XFD_ERR , -#endif -MSR_IA32_SYSENTER_CS , -MSR_IA32_SYSENTER_ESP , -MSR_IA32_SYSENTER_EIP , -MSR_CORE_C1_RES , -MSR_CORE_C3_RESIDENCY , -MSR_CORE_C6_RESIDENCY , -MSR_CORE_C7_RESIDENCY , -}; -/* * These 2 parameters are used to config the controls for Pause-Loop Exiting: * ple_gap: upper bound on the amount of time between two successive
@@ -672,40 +647,6 @@ static inline bool cpu_need_virtualize_apic_accesses(struct kvm_vcpu *vcpu)return flexpriority_enabled &&lapic_in_kernel (vcpu); } -static int vmx_get_passthrough_msr_slot (u32 msr) -{ -int i; - -switch (msr) { - case 0x800 ... 0x8ff: -/* x2APIC MSRs. These are handled in vmx_update_msr_bitmap_x2apic() */ -return -ENOENT ; -case MSR_IA32_RTIT_STATUS : -case MSR_IA32_RTIT_OUTPUT_BASE : -case MSR_IA32_RTIT_OUTPUT_MASK : -case MSR_IA32_RTIT_CR3_MATCH : - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: -/* PT MSRs. These are handled in pt_update_intercept_for_msr() */ -case MSR_LBR_SELECT : -case MSR_LBR_TOS : - case MSR_LBR_INFO_0 ... MSR_LBR_INFO_0 + 31: - case MSR_LBR_NHM_FROM ... MSR_LBR_NHM_FROM + 31: - case MSR_LBR_NHM_TO ... MSR_LBR_NHM_TO + 31: - case MSR_LBR_CORE_FROM ... MSR_LBR_CORE_FROM + 8: - case MSR_LBR_CORE_TO ... MSR_LBR_CORE_TO + 8: -/* LBR MSRs. These are handled in vmx_update_intercept_for_lbr_msrs() */ -return -ENOENT ; - } - -for (i =0 ; i <ARRAY_SIZE (vmx_possible_passthrough_msrs); i++) { -if (vmx_possible_passthrough_msrs[i] == msr) -return i; - } - -WARN (1 ,"Invalid MSR %x, please adapt vmx_possible_passthrough_msrs[]" , msr); -return -ENOENT ; -} -struct vmx_uret_msr *vmx_find_uret_msr (struct vcpu_vmx *vmx,u32 msr) {int i;
@@ -4015,25 +3956,12 @@ void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) {struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned long *msr_bitmap = vmx->vmcs01.msr_bitmap; -int idx;if (!cpu_has_vmx_msr_bitmap ())return ;vmx_msr_bitmap_l01_changed (vmx); -/* -* Mark the desired intercept state in shadow bitmap, this is needed -* for resync when the MSR filters change. -*/ - idx =vmx_get_passthrough_msr_slot (msr); -if (idx >=0 ) { -if (type &MSR_TYPE_R ) -__clear_bit (idx, vmx->shadow_msr_intercept.read); -if (type &MSR_TYPE_W ) -__clear_bit (idx, vmx->shadow_msr_intercept.write); - } -if ((type &MSR_TYPE_R ) && !kvm_msr_allowed (vcpu, msr,KVM_MSR_FILTER_READ )) {vmx_set_msr_bitmap_read (msr_bitmap, msr);
@@ -4057,25 +3985,12 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) {struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned long *msr_bitmap = vmx->vmcs01.msr_bitmap; -int idx;if (!cpu_has_vmx_msr_bitmap ())return ;vmx_msr_bitmap_l01_changed (vmx); -/* -* Mark the desired intercept state in shadow bitmap, this is needed -* for resync when the MSR filter changes. -*/ - idx =vmx_get_passthrough_msr_slot (msr); -if (idx >=0 ) { -if (type &MSR_TYPE_R ) -__set_bit (idx, vmx->shadow_msr_intercept.read); -if (type &MSR_TYPE_W ) -__set_bit (idx, vmx->shadow_msr_intercept.write); - } -if (type &MSR_TYPE_R )vmx_set_msr_bitmap_read (msr_bitmap, msr);
@@ -4159,35 +4074,59 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) } } -void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) +static void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu) { -struct vcpu_vmx *vmx =to_vmx (vcpu); -u32 i; -if (!cpu_has_vmx_msr_bitmap ())return ; -/* -* Redo intercept permissions for MSRs that KVM is passing through to -* the guest. Disabling interception will check the new MSR filter and -* ensure that KVM enables interception if usersepace wants to filter -* the MSR. MSRs that KVM is already intercepting don't need to be -* refreshed since KVM is going to intercept them regardless of what -* userspace wants. -*/ -for (i =0 ; i <ARRAY_SIZE (vmx_possible_passthrough_msrs); i++) { -u32 msr = vmx_possible_passthrough_msrs[i]; - -if (!test_bit (i, vmx->shadow_msr_intercept.read)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_R ); - -if (!test_bit (i, vmx->shadow_msr_intercept.write)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_W ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_TSC ,MSR_TYPE_R ); +#ifdef CONFIG_X86_64 +vmx_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); +#endif +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_ESP ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_EIP ,MSR_TYPE_RW ); +if (kvm_cstate_in_guest (vcpu->kvm)) { +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C1_RES ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C3_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C6_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C7_RESIDENCY ,MSR_TYPE_R ); }/* PT MSRs can be passed through iff PT is exposed to the guest. */ if (vmx_pt_mode_is_host_guest ())pt_update_intercept_for_msr (vcpu); + +if (vcpu->arch.xfd_no_write_intercept) +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_XFD ,MSR_TYPE_RW ); + + +vmx_set_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW , + !to_vmx (vcpu)->spec_ctrl); + +if (kvm_cpu_cap_has (X86_FEATURE_XFD )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , + !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); + +if (boot_cpu_has (X86_FEATURE_IBPB )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); + +if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* x2APIC and LBR MSR intercepts are modified on-demand and cannot be +* filtered by userspace. +*/ +} + +void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) +{ +vmx_recalc_msr_intercepts (vcpu); } static int vmx_deliver_nested_posted_interrupt(struct kvm_vcpu *vcpu,
@@ -7537,26 +7476,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) evmcs->hv_enlightenments_control.msr_bitmap =1 ; } -/* The MSR bitmap starts with all ones */ -bitmap_fill (vmx->shadow_msr_intercept.read,MAX_POSSIBLE_PASSTHROUGH_MSRS ); -bitmap_fill (vmx->shadow_msr_intercept.write,MAX_POSSIBLE_PASSTHROUGH_MSRS ); - -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_TSC ,MSR_TYPE_R ); -#ifdef CONFIG_X86_64 -vmx_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); -#endif -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_ESP ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_EIP ,MSR_TYPE_RW ); -if (kvm_cstate_in_guest (vcpu->kvm)) { -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C1_RES ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C3_RESIDENCY ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C6_RESIDENCY ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C7_RESIDENCY ,MSR_TYPE_R ); - } - vmx->loaded_vmcs = &vmx->vmcs01;if (cpu_need_virtualize_apic_accesses (vcpu)) {
@@ -7842,18 +7761,6 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) } } -if (kvm_cpu_cap_has (X86_FEATURE_XFD )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , - !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); - -if (boot_cpu_has (X86_FEATURE_IBPB )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , - !guest_has_pred_cmd_msr (vcpu)); - -if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , - !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); -set_cr4_guest_host_mask (vmx);vmx_write_encls_bitmap (vcpu,NULL );
@@ -7869,6 +7776,9 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_LC_ENABLED ; +/* Recalc MSR interception to account for feature changes. */ +vmx_recalc_msr_intercepts (vcpu); +/* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap (vcpu); }
arch/x86/kvm/vmx/vmx.h +0 -7
@@ -294,13 +294,6 @@ struct vcpu_vmx {struct pt_desc pt_desc;struct lbr_desc lbr_desc; -/* Save desired MSR intercept (read: pass-through) state */ -#define MAX_POSSIBLE_PASSTHROUGH_MSRS 16 -struct { - DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS); - DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS); - } shadow_msr_intercept; -/* ve_info must be page aligned. */ struct vmx_ve_information *ve_info; };
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 16/28] KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter change
Quoted 9 lines · Sean Christopherson
On a userspace MSR filter change, recalculate all MSR intercepts using the filter-agnostic logic instead of maintaining a "shadow copy" of KVM's desired intercepts. The shadow bitmaps add yet another point of failure, are confusing (e.g. what does "handled specially" mean!?!?), an eyesore, and a maintenance burden.
Given that KVM *must* be able to recalculate the correct intercepts at any given time, and that MSR filter updates are not hot paths, there is zero benefit to maintaining the shadow bitmaps.
+1
To me, this patch does simplify the logic by removing the bitmap state management.
Just one very minor comment below — other than that:
Quoted 55 lines
arch/x86/kvm/vmx/vmx.c +18
-20@@ -4159,35 +4074,59 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) } } -void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) +static void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu) { -struct vcpu_vmx *vmx =to_vmx (vcpu); -u32 i; -if (!cpu_has_vmx_msr_bitmap ())return ; -/* -* Redo intercept permissions for MSRs that KVM is passing through to -* the guest. Disabling interception will check the new MSR filter and -* ensure that KVM enables interception if usersepace wants to filter -* the MSR. MSRs that KVM is already intercepting don't need to be -* refreshed since KVM is going to intercept them regardless of what -* userspace wants. -*/ -for (i =0 ; i <ARRAY_SIZE (vmx_possible_passthrough_msrs); i++) { -u32 msr = vmx_possible_passthrough_msrs[i]; - -if (!test_bit (i, vmx->shadow_msr_intercept.read)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_R ); - -if (!test_bit (i, vmx->shadow_msr_intercept.write)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_W ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_TSC ,MSR_TYPE_R ); +#ifdef CONFIG_X86_64 +vmx_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); +#endif +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_ESP ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_EIP ,MSR_TYPE_RW ); +if (kvm_cstate_in_guest (vcpu->kvm)) { +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C1_RES ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C3_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C6_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C7_RESIDENCY ,MSR_TYPE_R ); }/* PT MSRs can be passed through iff PT is exposed to the guest. */ if (vmx_pt_mode_is_host_guest ())pt_update_intercept_for_msr (vcpu); + +if (vcpu->arch.xfd_no_write_intercept) +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_XFD ,MSR_TYPE_RW );
arch/x86/kvm/vmx/vmx.c void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) · line 4113
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10
-0+ + +vmx_set_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW , + !to_vmx (vcpu)->spec_ctrl); + +if (kvm_cpu_cap_has (X86_FEATURE_XFD )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , + !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); + +if (boot_cpu_has (X86_FEATURE_IBPB ))
I think Boris prefers using cpu_feature_enabled() instead — maybe this is a good opportunity to update this occurrence?
arch/x86/kvm/vmx/vmx.c void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) · line 4117
Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4
-0+vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); + +if (boot_cpu_has (X86_FEATURE_FLUSH_L1D ))
Ditto.
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +8
-0+vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* x2APIC and LBR MSR intercepts are modified on-demand and cannot be +* filtered by userspace. +*/ +}
Sean Christopherson
maintainerRe: [PATCH 16/28] KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter change
Quoted 8 lines · Xin Li
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +8
-0+ +if (vcpu->arch.xfd_no_write_intercept) +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_XFD ,MSR_TYPE_RW ); + + +vmx_set_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW , + !to_vmx (vcpu)->spec_ctrl); +
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) · line 4113
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +5
-0+if (kvm_cpu_cap_has (X86_FEATURE_XFD )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , + !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); + +if (boot_cpu_has (X86_FEATURE_IBPB ))I think Boris prefers using
cpu_feature_enabled()instead — maybe this is a good opportunity to update this occurrence?
Yeah, I'm comfortable squeezing in that change.
Quoted 15 lines
arch/x86/kvm/vmx/vmx.c void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) · line 4117
Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4
-0+vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); + +if (boot_cpu_has (X86_FEATURE_FLUSH_L1D ))Ditto.
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +8
-0+vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* x2APIC and LBR MSR intercepts are modified on-demand and cannot be +* filtered by userspace. +*/ +}
Re: [PATCH 16/28] KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter change
In reply to Sean ChristophersonQuoted 112 lines · Sean Christopherson
On a userspace MSR filter change, recalculate all MSR intercepts using the filter-agnostic logic instead of maintaining a "shadow copy" of KVM's desired intercepts. The shadow bitmaps add yet another point of failure, are confusing (e.g. what does "handled specially" mean!?!?), an eyesore, and a maintenance burden.
Given that KVM *must* be able to recalculate the correct intercepts at any given time, and that MSR filter updates are not hot paths, there is zero benefit to maintaining the shadow bitmaps.
Cc Borislav Petkov <[email protected]>Cc Xin Li <[email protected]>Cc Chao Gao <[email protected]>Cc Dapeng Mi <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/vmx.c | 184 +++++++++++------------------------------ arch/x86/kvm/vmx/vmx.h | 7 -- 2 files changed, 47 insertions(+), 144 deletions(-)arch/x86/kvm/vmx/vmx.c +0
-62@@ -166,31 +166,6 @@ module_param(allow_smaller_maxphyaddr, bool, S_IRUGO);RTIT_STATUS_ERROR |RTIT_STATUS_STOPPED | \ RTIT_STATUS_BYTECNT)) -/* -* List of MSRs that can be directly passed to the guest. -* In addition to these x2apic, PT and LBR MSRs are handled specially. -*/ -static u32 vmx_possible_passthrough_msrs[MAX_POSSIBLE_PASSTHROUGH_MSRS ] = { -MSR_IA32_SPEC_CTRL , -MSR_IA32_PRED_CMD , -MSR_IA32_FLUSH_CMD , -MSR_IA32_TSC , -#ifdef CONFIG_X86_64 -MSR_FS_BASE , -MSR_GS_BASE , -MSR_KERNEL_GS_BASE , -MSR_IA32_XFD , -MSR_IA32_XFD_ERR , -#endif -MSR_IA32_SYSENTER_CS , -MSR_IA32_SYSENTER_ESP , -MSR_IA32_SYSENTER_EIP , -MSR_CORE_C1_RES , -MSR_CORE_C3_RESIDENCY , -MSR_CORE_C6_RESIDENCY , -MSR_CORE_C7_RESIDENCY , -}; -/* * These 2 parameters are used to config the controls for Pause-Loop Exiting: * ple_gap: upper bound on the amount of time between two successive @@ -672,40 +647,6 @@ static inline bool cpu_need_virtualize_apic_accesses(struct kvm_vcpu *vcpu)return flexpriority_enabled &&lapic_in_kernel (vcpu); } -static int vmx_get_passthrough_msr_slot (u32 msr) -{ -int i; - -switch (msr) { - case 0x800 ... 0x8ff: -/* x2APIC MSRs. These are handled in vmx_update_msr_bitmap_x2apic() */ -return -ENOENT ; -case MSR_IA32_RTIT_STATUS : -case MSR_IA32_RTIT_OUTPUT_BASE : -case MSR_IA32_RTIT_OUTPUT_MASK : -case MSR_IA32_RTIT_CR3_MATCH : - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: -/* PT MSRs. These are handled in pt_update_intercept_for_msr() */ -case MSR_LBR_SELECT : -case MSR_LBR_TOS : - case MSR_LBR_INFO_0 ... MSR_LBR_INFO_0 + 31: - case MSR_LBR_NHM_FROM ... MSR_LBR_NHM_FROM + 31: - case MSR_LBR_NHM_TO ... MSR_LBR_NHM_TO + 31: - case MSR_LBR_CORE_FROM ... MSR_LBR_CORE_FROM + 8: - case MSR_LBR_CORE_TO ... MSR_LBR_CORE_TO + 8: -/* LBR MSRs. These are handled in vmx_update_intercept_for_lbr_msrs() */ -return -ENOENT ; - } - -for (i =0 ; i <ARRAY_SIZE (vmx_possible_passthrough_msrs); i++) { -if (vmx_possible_passthrough_msrs[i] == msr) -return i; - } - -WARN (1 ,"Invalid MSR %x, please adapt vmx_possible_passthrough_msrs[]" , msr); -return -ENOENT ; -} -struct vmx_uret_msr *vmx_find_uret_msr (struct vcpu_vmx *vmx,u32 msr) {int i;@@ -4015,25 +3956,12 @@ void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) {struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned long *msr_bitmap = vmx->vmcs01.msr_bitmap; -int idx;if (!cpu_has_vmx_msr_bitmap ())return ;vmx_msr_bitmap_l01_changed (vmx); -/* -* Mark the desired intercept state in shadow bitmap, this is needed
arch/x86/kvm/vmx/vmx.c void vmx_disable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) · line 4036
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +0
-10-* for resync when the MSR filters change. -*/ - idx =vmx_get_passthrough_msr_slot (msr); -if (idx >=0 ) { -if (type &MSR_TYPE_R ) -__clear_bit (idx, vmx->shadow_msr_intercept.read); -if (type &MSR_TYPE_W ) -__clear_bit (idx, vmx->shadow_msr_intercept.write); - } -
The patch looks good to me. Only a minor code refine on vmx_disable_intercept_for_msr().
arch/x86/kvm/vmx/vmx.c +0 -0
@@ -4041,23 +4041,19 @@ void vmx_disable_intercept_for_msr(struct kvm_vcpu
*vcpu, u32 msr, int type)
clear_bit(idx, vmx->shadow_msr_intercept.write); }
- if ((type & MSR_TYPE_R) &&
- !kvm_msr_allowed(vcpu, msr, KVM_MSR_FILTER_READ)) { - vmx_set_msr_bitmap_read(msr_bitmap, msr); - type &= ~MSR_TYPE_R;
+ if (type & MSR_TYPE_R) {
+ if (!kvm_msr_allowed(vcpu, msr, KVM_MSR_FILTER_READ)) + vmx_set_msr_bitmap_read(msr_bitmap, msr); + else
+ vmx_clear_msr_bitmap_read(msr_bitmap, msr); }
- if ((type & MSR_TYPE_W) &&
- !kvm_msr_allowed(vcpu, msr, KVM_MSR_FILTER_WRITE)) { - vmx_set_msr_bitmap_write(msr_bitmap, msr); - type &= ~MSR_TYPE_W;
+ if (type & MSR_TYPE_W) {
+ if (!kvm_msr_allowed(vcpu, msr, KVM_MSR_FILTER_WRITE)) + vmx_set_msr_bitmap_write(msr_bitmap, msr); + else
+ vmx_clear_msr_bitmap_write(msr_bitmap, msr); }
-
- if (type & MSR_TYPE_R)
- vmx_clear_msr_bitmap_read(msr_bitmap, msr); -
- if (type & MSR_TYPE_W)
- vmx_clear_msr_bitmap_write(msr_bitmap, msr); }
It looks simpler and easily understood.
Quoted 183 lines
arch/x86/kvm/vmx/vmx.c +47
-65if ((type &MSR_TYPE_R ) && !kvm_msr_allowed (vcpu, msr,KVM_MSR_FILTER_READ )) {vmx_set_msr_bitmap_read (msr_bitmap, msr);@@ -4057,25 +3985,12 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) {struct vcpu_vmx *vmx =to_vmx (vcpu);unsigned long *msr_bitmap = vmx->vmcs01.msr_bitmap; -int idx;if (!cpu_has_vmx_msr_bitmap ())return ;vmx_msr_bitmap_l01_changed (vmx); -/* -* Mark the desired intercept state in shadow bitmap, this is needed -* for resync when the MSR filter changes. -*/ - idx =vmx_get_passthrough_msr_slot (msr); -if (idx >=0 ) { -if (type &MSR_TYPE_R ) -__set_bit (idx, vmx->shadow_msr_intercept.read); -if (type &MSR_TYPE_W ) -__set_bit (idx, vmx->shadow_msr_intercept.write); - } -if (type &MSR_TYPE_R )vmx_set_msr_bitmap_read (msr_bitmap, msr);@@ -4159,35 +4074,59 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) } } -void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) +static void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu) { -struct vcpu_vmx *vmx =to_vmx (vcpu); -u32 i; -if (!cpu_has_vmx_msr_bitmap ())return ; -/* -* Redo intercept permissions for MSRs that KVM is passing through to -* the guest. Disabling interception will check the new MSR filter and -* ensure that KVM enables interception if usersepace wants to filter -* the MSR. MSRs that KVM is already intercepting don't need to be -* refreshed since KVM is going to intercept them regardless of what -* userspace wants. -*/ -for (i =0 ; i <ARRAY_SIZE (vmx_possible_passthrough_msrs); i++) { -u32 msr = vmx_possible_passthrough_msrs[i]; - -if (!test_bit (i, vmx->shadow_msr_intercept.read)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_R ); - -if (!test_bit (i, vmx->shadow_msr_intercept.write)) -vmx_disable_intercept_for_msr (vcpu, msr,MSR_TYPE_W ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_TSC ,MSR_TYPE_R ); +#ifdef CONFIG_X86_64 +vmx_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); +#endif +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_ESP ,MSR_TYPE_RW ); +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_EIP ,MSR_TYPE_RW ); +if (kvm_cstate_in_guest (vcpu->kvm)) { +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C1_RES ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C3_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C6_RESIDENCY ,MSR_TYPE_R ); +vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C7_RESIDENCY ,MSR_TYPE_R ); }/* PT MSRs can be passed through iff PT is exposed to the guest. */ if (vmx_pt_mode_is_host_guest ())pt_update_intercept_for_msr (vcpu); + +if (vcpu->arch.xfd_no_write_intercept) +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_XFD ,MSR_TYPE_RW ); + + +vmx_set_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW , + !to_vmx (vcpu)->spec_ctrl); + +if (kvm_cpu_cap_has (X86_FEATURE_XFD )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , + !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); + +if (boot_cpu_has (X86_FEATURE_IBPB )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); + +if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) +vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* x2APIC and LBR MSR intercepts are modified on-demand and cannot be +* filtered by userspace. +*/ +} + +void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) +{ +vmx_recalc_msr_intercepts (vcpu); } static int vmx_deliver_nested_posted_interrupt(struct kvm_vcpu *vcpu,@@ -7537,26 +7476,6 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) evmcs->hv_enlightenments_control.msr_bitmap =1 ; } -/* The MSR bitmap starts with all ones */ -bitmap_fill (vmx->shadow_msr_intercept.read,MAX_POSSIBLE_PASSTHROUGH_MSRS ); -bitmap_fill (vmx->shadow_msr_intercept.write,MAX_POSSIBLE_PASSTHROUGH_MSRS ); - -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_TSC ,MSR_TYPE_R ); -#ifdef CONFIG_X86_64 -vmx_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); -#endif -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_ESP ,MSR_TYPE_RW ); -vmx_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_EIP ,MSR_TYPE_RW ); -if (kvm_cstate_in_guest (vcpu->kvm)) { -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C1_RES ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C3_RESIDENCY ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C6_RESIDENCY ,MSR_TYPE_R ); -vmx_disable_intercept_for_msr (vcpu,MSR_CORE_C7_RESIDENCY ,MSR_TYPE_R ); - } - vmx->loaded_vmcs = &vmx->vmcs01;if (cpu_need_virtualize_apic_accesses (vcpu)) {@@ -7842,18 +7761,6 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) } } -if (kvm_cpu_cap_has (X86_FEATURE_XFD )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_XFD_ERR ,MSR_TYPE_R , - !guest_cpu_cap_has (vcpu,X86_FEATURE_XFD )); - -if (boot_cpu_has (X86_FEATURE_IBPB )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , - !guest_has_pred_cmd_msr (vcpu)); - -if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) -vmx_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , - !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); -set_cr4_guest_host_mask (vmx);vmx_write_encls_bitmap (vcpu,NULL );@@ -7869,6 +7776,9 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_LC_ENABLED ; +/* Recalc MSR interception to account for feature changes. */ +vmx_recalc_msr_intercepts (vcpu); +/* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap (vcpu); }arch/x86/kvm/vmx/vmx.h +0
-7@@ -294,13 +294,6 @@ struct vcpu_vmx {struct pt_desc pt_desc;struct lbr_desc lbr_desc; -/* Save desired MSR intercept (read: pass-through) state */ -#define MAX_POSSIBLE_PASSTHROUGH_MSRS 16 -struct { - DECLARE_BITMAP(read, MAX_POSSIBLE_PASSTHROUGH_MSRS); - DECLARE_BITMAP(write, MAX_POSSIBLE_PASSTHROUGH_MSRS); - } shadow_msr_intercept; -/* ve_info must be page aligned. */ struct vmx_ve_information *ve_info; };
Re: [PATCH 16/28] KVM: VMX: Manually recalc all MSR intercepts on userspace MSR filter change
In reply to Sean ChristophersonQuoted 10 lines · Sean Christopherson
On a userspace MSR filter change, recalculate all MSR intercepts using the filter-agnostic logic instead of maintaining a "shadow copy" of KVM's desired intercepts. The shadow bitmaps add yet another point of failure, are confusing (e.g. what does "handled specially" mean!?!?), an eyesore, and a maintenance burden.
Given that KVM *must* be able to recalculate the correct intercepts at any given time, and that MSR filter updates are not hot paths, there is zero benefit to maintaining the shadow bitmaps.
Quoted 7 lines
Cc Borislav Petkov <[email protected]>Cc Xin Li <[email protected]>Cc Chao Gao <[email protected]>Cc Dapeng Mi <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
one nit below,
arch/x86/kvm/vmx/vmx.c void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) · line 4105
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +5
-0+ +if (vcpu->arch.xfd_no_write_intercept) +vmx_disable_intercept_for_msr (vcpu,MSR_IA32_XFD ,MSR_TYPE_RW ); + +
Remove one newline here.
Sean Christopherson
maintainerPATCH 17/28 KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change
In reply to Sean ChristophersonOn a userspace MSR filter change, recalculate all MSR intercepts using the filter-agnostic logic instead of maintaining a "shadow copy" of KVM's desired intercepts. The shadow bitmaps add yet another point of failure, are confusing (e.g. what does "handled specially" mean!?!?), an eyesore, and a maintenance burden.
Given that KVM *must* be able to recalculate the correct intercepts at any given time, and that MSR filter updates are not hot paths, there is zero benefit to maintaining the shadow bitmaps.
--- arch/x86/kvm/svm/sev.c | 16 +- arch/x86/kvm/svm/svm.c | 371 +++++++++++------------------------------ arch/x86/kvm/svm/svm.h | 7 +- 3 files changed, 105 insertions(+), 289 deletions(-)
arch/x86/kvm/svm/sev.c +5 -11
@@ -4411,9 +4411,12 @@ int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in) count, in); } -static void sev_es_vcpu_after_set_cpuid (struct vcpu_svm *svm) +void sev_es_recalc_msr_intercepts (struct kvm_vcpu *vcpu) { -struct kvm_vcpu *vcpu = &svm->vcpu; +/* Clear intercepts on MSRs that are context switched by hardware. */ +svm_disable_intercept_for_msr (vcpu,MSR_AMD64_SEV_ES_GHCB ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_EFER ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_IA32_CR_PAT ,MSR_TYPE_RW );if (boot_cpu_has (X86_FEATURE_V_TSC_AUX )) svm_set_intercept_for_msr(vcpu, MSR_TSC_AUX, MSR_TYPE_RW,
@@ -4448,16 +4451,12 @@ void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm) best =kvm_find_cpuid_entry (vcpu,0x8000001F );if (best) vcpu->arch.reserved_gpa_bits &= ~(1UL << (best->ebx &0x3f )); - -if (sev_es_guest (svm->vcpu.kvm)) -sev_es_vcpu_after_set_cpuid (svm); }static void sev_es_init_vmcb (struct vcpu_svm *svm) {struct kvm_sev_info *sev =to_kvm_sev_info (svm->vcpu.kvm);struct vmcb *vmcb = svm->vmcb01.ptr; -struct kvm_vcpu *vcpu = &svm->vcpu; svm->vmcb->control.nested_ctl |=SVM_NESTED_CTL_SEV_ES_ENABLE ;
@@ -4511,11 +4510,6 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm)/* Can't intercept XSETBV, HV can't modify XCR0 directly */ svm_clr_intercept (svm,INTERCEPT_XSETBV ); - -/* Clear intercepts on MSRs that are context switched by hardware. */ -svm_disable_intercept_for_msr (vcpu,MSR_AMD64_SEV_ES_GHCB ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_EFER ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_IA32_CR_PAT ,MSR_TYPE_RW ); }void sev_init_vmcb (struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.c +99 -272
490 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +1 -6
@@ -318,12 +318,6 @@ struct vcpu_svm {struct list_head ir_list;spinlock_t ir_list_lock; -/* Save desired MSR intercept (read: pass-through) state */ -struct { - DECLARE_BITMAP(read, MAX_DIRECT_ACCESS_MSRS); - DECLARE_BITMAP(write, MAX_DIRECT_ACCESS_MSRS); - } shadow_msr_intercept; -struct vcpu_sev_es_state sev_es;bool guest_state_loaded;
@@ -824,6 +818,7 @@ void sev_init_vmcb(struct vcpu_svm *svm);void sev_vcpu_after_set_cpuid (struct vcpu_svm *svm);int sev_es_string_io (struct vcpu_svm *svm,int size,unsigned int port,int in);void sev_es_vcpu_reset (struct vcpu_svm *svm); +void sev_es_recalc_msr_intercepts (struct kvm_vcpu *vcpu);void sev_vcpu_deliver_sipi_vector (struct kvm_vcpu *vcpu,u8 vector);void sev_es_prepare_switch_to_guest (struct vcpu_svm *svm,struct sev_es_save_area *hostsa);void sev_es_unmap_ghcb (struct vcpu_svm *svm);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 17/28] KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change
Quoted 59 lines · Sean Christopherson
arch/x86/kvm/svm/svm.c +0
-46@@ -81,70 +79,6 @@ static uint64_t osvw_len = 4, osvw_status; static DEFINE_PER_CPU(u64, current_tsc_ratio); -static const u32 direct_access_msrs[] = { -MSR_STAR , -MSR_IA32_SYSENTER_CS , -MSR_IA32_SYSENTER_EIP , -MSR_IA32_SYSENTER_ESP , -#ifdef CONFIG_X86_64 -MSR_GS_BASE , -MSR_FS_BASE , -MSR_KERNEL_GS_BASE , -MSR_LSTAR , -MSR_CSTAR , -MSR_SYSCALL_MASK , -#endif -MSR_IA32_SPEC_CTRL , -MSR_IA32_PRED_CMD , -MSR_IA32_FLUSH_CMD , -MSR_IA32_DEBUGCTLMSR , -MSR_IA32_LASTBRANCHFROMIP , -MSR_IA32_LASTBRANCHTOIP , -MSR_IA32_LASTINTFROMIP , -MSR_IA32_LASTINTTOIP , -MSR_IA32_XSS , -MSR_EFER , -MSR_IA32_CR_PAT , -MSR_AMD64_SEV_ES_GHCB , -MSR_TSC_AUX , -X2APIC_MSR (APIC_ID ), -X2APIC_MSR (APIC_LVR ), -X2APIC_MSR (APIC_TASKPRI ), -X2APIC_MSR (APIC_ARBPRI ), -X2APIC_MSR (APIC_PROCPRI ), -X2APIC_MSR (APIC_EOI ), -X2APIC_MSR (APIC_RRR ), -X2APIC_MSR (APIC_LDR ), -X2APIC_MSR (APIC_DFR ), -X2APIC_MSR (APIC_SPIV ), -X2APIC_MSR (APIC_ISR ), -X2APIC_MSR (APIC_TMR ), -X2APIC_MSR (APIC_IRR ), -X2APIC_MSR (APIC_ESR ), -X2APIC_MSR (APIC_ICR ), -X2APIC_MSR (APIC_ICR2 ), - -/* -* Note: -* AMD does not virtualize APIC TSC-deadline timer mode, but it is
(quoted diff) +0
-1- * emulated by KVM. When setting APIC LVTT (0x832) register bit18,
arch/x86/kvm/svm/svm.c +0
-6-* the AVIC hardware would generate GP fault. Therefore, always -* intercept the MSR 0x832, and do not setup direct_access_msr. -*/ -X2APIC_MSR (APIC_LVTTHMR ), -X2APIC_MSR (APIC_LVTPC ), - X2APIC_MSR(APIC_LVT0),
arch/x86/kvm/svm/svm.c static uint64_t osvw_len = 4, osvw_status; · line 146
Quoted 10 lines
arch/x86/kvm/svm/svm.c +0
-10-X2APIC_MSR (APIC_LVT1 ), -X2APIC_MSR (APIC_LVTERR ), -X2APIC_MSR (APIC_TMICT ), -X2APIC_MSR (APIC_TMCCT ), -X2APIC_MSR (APIC_TDCR ), -}; - -static_assert (ARRAY_SIZE (direct_access_msrs) == -MAX_DIRECT_ACCESS_MSRS -6 * !IS_ENABLED (CONFIG_X86_64 )); -#undef MAX_DIRECT_ACCESS_MSRS
The MAX_DIRECT_ACCESS_MSRS define should now be removed from arch/x86/kvm/svm/svm.harch/x86/kvm/svm/svm.h, since it's no longer used.
Sean Christopherson
maintainerRe: [PATCH 17/28] KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change
Quoted 7 lines · Francesco Lavra
arch/x86/kvm/svm/svm.c static uint64_t osvw_len = 4, osvw_status; · line 146
Quoted 3 lines · Sean Christopherson
arch/x86/kvm/svm/svm.c +0
-3-static_assert (ARRAY_SIZE (direct_access_msrs) == -MAX_DIRECT_ACCESS_MSRS -6 * !IS_ENABLED (CONFIG_X86_64 )); -#undef MAX_DIRECT_ACCESS_MSRSThe
MAX_DIRECT_ACCESS_MSRSdefine should now be removed fromarch/x86/kvm/svm/svm.harch/x86/kvm/svm/svm.h, since it's no longer used.
/facepalm
All that work to get rid of the darn thing, and then I forget to actually get rid of it.
I also failed to remove the msrpm_offset declaration (and its size macro)
#define MSRPM_OFFSETS 32 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly;
which are dead code as of "KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change".
Nice catch, and thank you!
Re: [PATCH 17/28] KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change
In reply to Sean ChristophersonQuoted 13 lines
arch/x86/kvm/svm/svm.c +13
-0+static void svm_recalc_msr_intercepts (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); + +svm_vcpu_init_msrpm (vcpu); + +if (lbrv) +svm_recalc_lbr_msr_intercepts (vcpu); + +if (boot_cpu_has (X86_FEATURE_IBPB )) +svm_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); +
arch/x86/kvm/svm/svm.c void svm_vcpu_free_msrpm(u32 *msrpm) · line 881
Quoted 10 lines
arch/x86/kvm/svm/svm.c +10
-0+if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) +svm_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* Unconditionally disable interception of SPEC_CTRL if V_SPEC_CTRL is +* supported, i.e. if VMRUN/#VMEXIT context switch MSR_IA32_SPEC_CTRL. +*/ +if (boot_cpu_has (X86_FEATURE_V_SPEC_CTRL )) +svm_disable_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW );
I think there is a bug in the original code. KVM should inject #GP when guests try to access unsupported MSRs. Specifically, a guest w/o spec_ctrl support should get #GP when it tries to access the MSR regardless of V_SPEC_CTRL support on the host.
So, here should be
if (boot_cpu_has(X86_FEATURE_V_SPEC_CTRL)) svm_set_intercept_for_msr(vcpu, MSR_IA32_SPEC_CTRL, MSR_TYPE_RW, !guest_has_spec_ctrl_msr(vcpu));
Sean Christopherson
maintainerRe: [PATCH 17/28] KVM: SVM: Manually recalc all MSR intercepts on userspace MSR filter change
Quoted 13 lines · Chao Gao
Quoted 13 lines
arch/x86/kvm/svm/svm.c +13
-0+static void svm_recalc_msr_intercepts (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); + +svm_vcpu_init_msrpm (vcpu); + +if (lbrv) +svm_recalc_lbr_msr_intercepts (vcpu); + +if (boot_cpu_has (X86_FEATURE_IBPB )) +svm_set_intercept_for_msr (vcpu,MSR_IA32_PRED_CMD ,MSR_TYPE_W , + !guest_has_pred_cmd_msr (vcpu)); +
Quoted 8 lines
arch/x86/kvm/svm/svm.c void svm_vcpu_free_msrpm(u32 *msrpm) · line 879
Quoted 8 lines
arch/x86/kvm/svm/svm.c +8
-0+if (boot_cpu_has (X86_FEATURE_FLUSH_L1D )) +svm_set_intercept_for_msr (vcpu,MSR_IA32_FLUSH_CMD ,MSR_TYPE_W , + !guest_cpu_cap_has (vcpu,X86_FEATURE_FLUSH_L1D )); + +/* +* Unconditionally disable interception of SPEC_CTRL if V_SPEC_CTRL is +* supported, i.e. if VMRUN/#VMEXIT context switch MSR_IA32_SPEC_CTRL. +*/
Tangentially related, the comment above isn't quite correct. MSR_IA32_SPEC_CTRL isn't purely context switched, the CPU merges together host and guest values. Same end result though: KVM doesn't need to manually context switch the MSR.
Quoted 8 lines
arch/x86/kvm/svm/svm.c void svm_vcpu_free_msrpm(u32 *msrpm) · line 881
Quoted 2 lines
arch/x86/kvm/svm/svm.c +2
-0+if (boot_cpu_has (X86_FEATURE_V_SPEC_CTRL )) +svm_disable_intercept_for_msr (vcpu,MSR_IA32_SPEC_CTRL ,MSR_TYPE_RW );I think there is a bug in the original code. KVM should inject #GP when guests try to access unsupported MSRs. Specifically, a guest w/o
spec_ctrlsupport should get #GP when it tries to access the MSR regardless ofV_SPEC_CTRLsupport on the host.
Quoted 5 lines
So, here should be
if (boot_cpu_has(X86_FEATURE_V_SPEC_CTRL)) svm_set_intercept_for_msr(vcpu, MSR_IA32_SPEC_CTRL, MSR_TYPE_RW, !guest_has_spec_ctrl_msr(vcpu));
Right you are! I'll slot in a patch earlier in the series, and then it'll end up looking like this.
I'll also post a KUT testcase.
Sean Christopherson
maintainerPATCH 18/28 KVM: x86: Rename msr_filter_changed() => recalc_msr_intercepts()
In reply to Sean ChristophersonRename msr_filter_changed() to recalc_msr_intercepts() and drop the trampoline wrapper now that both SVM and VMX use a filter-agnostic recalc helper to react to the new userspace filter.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 8 +------- arch/x86/kvm/vmx/main.c | 6 +++--- arch/x86/kvm/vmx/vmx.c | 7 +------ arch/x86/kvm/vmx/x86_ops.h | 2 +- arch/x86/kvm/x86.c | 8 +++++++- 7 files changed, 15 insertions(+), 20 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -1
@@ -139,7 +139,7 @@ KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked) KVM_X86_OP_OPTIONAL(enable_l2_tlb_flush) KVM_X86_OP_OPTIONAL(migrate_timers) -KVM_X86_OP(msr_filter_changed) +KVM_X86_OP(recalc_msr_intercepts) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1885,7 +1885,7 @@ struct kvm_x86_ops {int (*enable_l2_tlb_flush)(struct kvm_vcpu *vcpu);void (*migrate_timers)(struct kvm_vcpu *vcpu); -void (*msr_filter_changed)(struct kvm_vcpu *vcpu); +void (*recalc_msr_intercepts)(struct kvm_vcpu *vcpu);int (*complete_emulated_msr)(struct kvm_vcpu *vcpu,int err);void (*vcpu_deliver_sipi_vector)(struct kvm_vcpu *vcpu,u8 vector);
arch/x86/kvm/svm/svm.c +1 -7
@@ -900,11 +900,6 @@ static void svm_recalc_msr_intercepts(struct kvm_vcpu *vcpu)*/ } -static void svm_msr_filter_changed (struct kvm_vcpu *vcpu) -{ -svm_recalc_msr_intercepts (vcpu); -} -void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb) { to_vmcb->save.dbgctl = from_vmcb->save.dbgctl;
@@ -933,7 +928,6 @@ static void svm_disable_lbrv(struct kvm_vcpu *vcpu)struct vcpu_svm *svm =to_svm (vcpu);KVM_BUG_ON (sev_es_guest (vcpu->kvm), vcpu->kvm); - svm->vmcb->control.virt_ext &= ~LBR_CTL_ENABLE_MASK ;svm_recalc_lbr_msr_intercepts (vcpu);
@@ -5231,7 +5225,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .apic_init_signal_blocked = svm_apic_init_signal_blocked, - .msr_filter_changed = svm_msr_filter_changed, + .recalc_msr_intercepts = svm_recalc_msr_intercepts, .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
arch/x86/kvm/vmx/main.c +3 -3
@@ -220,7 +220,7 @@ static int vt_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)return vmx_get_msr (vcpu, msr_info); } -static void vt_msr_filter_changed (struct kvm_vcpu *vcpu) +static void vt_recalc_msr_intercepts (struct kvm_vcpu *vcpu) {/* * TDX doesn't allow VMM to configure interception of MSR accesses.
@@ -231,7 +231,7 @@ static void vt_msr_filter_changed(struct kvm_vcpu *vcpu)if (is_td_vcpu (vcpu))return ; -vmx_msr_filter_changed (vcpu); +vmx_recalc_msr_intercepts (vcpu); }static int vt_complete_emulated_msr (struct kvm_vcpu *vcpu,int err)
@@ -1034,7 +1034,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .apic_init_signal_blocked = vt_op(apic_init_signal_blocked), .migrate_timers = vmx_migrate_timers, - .msr_filter_changed = vt_op(msr_filter_changed), + .recalc_msr_intercepts = vt_op(recalc_msr_intercepts), .complete_emulated_msr = vt_op(complete_emulated_msr), .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector,
arch/x86/kvm/vmx/vmx.c +1 -6
@@ -4074,7 +4074,7 @@ void pt_update_intercept_for_msr(struct kvm_vcpu *vcpu) } } -static void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu) +void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu) {if (!cpu_has_vmx_msr_bitmap ())return ;
@@ -4124,11 +4124,6 @@ static void vmx_recalc_msr_intercepts(struct kvm_vcpu *vcpu)*/ } -void vmx_msr_filter_changed (struct kvm_vcpu *vcpu) -{ -vmx_recalc_msr_intercepts (vcpu); -} -static int vmx_deliver_nested_posted_interrupt (struct kvm_vcpu *vcpu,int vector) {
arch/x86/kvm/vmx/x86_ops.h +1 -1
@@ -52,7 +52,7 @@ void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, int trig_mode, int vector);void vmx_vcpu_after_set_cpuid (struct kvm_vcpu *vcpu);bool vmx_has_emulated_msr (struct kvm *kvm,u32 index); -void vmx_msr_filter_changed (struct kvm_vcpu *vcpu); +void vmx_recalc_msr_intercepts (struct kvm_vcpu *vcpu);void vmx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void vmx_update_exception_bitmap (struct kvm_vcpu *vcpu);int vmx_get_feature_msr (u32 msr,u64 *data);
arch/x86/kvm/x86.c +7 -1
@@ -10924,8 +10924,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_vcpu_update_apicv (vcpu);if (kvm_check_request (KVM_REQ_APF_READY , vcpu))kvm_check_async_pf_completion (vcpu); + +/* +* Recalc MSR intercepts as userspace may want to intercept +* accesses to MSRs that KVM would otherwise pass through to +* the guest. +*/ if (kvm_check_request (KVM_REQ_MSR_FILTER_CHANGED , vcpu)) -kvm_x86_call (msr_filter_changed)(vcpu); +kvm_x86_call (recalc_msr_intercepts)(vcpu);if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu))kvm_x86_call (update_cpu_dirty_logging)(vcpu);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 18/28] KVM: x86: Rename msr_filter_changed() => recalc_msr_intercepts()
Quoted 7 lines · Sean Christopherson
Rename
msr_filter_changed()torecalc_msr_intercepts()and drop the trampoline wrapper now that both SVM and VMX use a filter-agnostic recalc helper to react to the new userspace filter.No functional change intended.
Signed-off-by Sean Christopherson<[email protected]>
Quoted 9 lines
--- arch/x86/include/asm/kvm-x86-ops.h | 2 +- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 8 +------- arch/x86/kvm/vmx/main.c | 6 +++--- arch/x86/kvm/vmx/vmx.c | 7 +------ arch/x86/kvm/vmx/x86_ops.h | 2 +- arch/x86/kvm/x86.c | 8 +++++++- 7 files changed, 15 insertions(+), 20 deletions(-)
Sean Christopherson
maintainerPATCH 19/28 KVM: SVM: Rename init_vmcb_after_set_cpuid() to make it intercepts specific
In reply to Sean ChristophersonRename init_vmcb_after_set_cpuid() to svm_recalc_intercepts_after_set_cpuid() to more precisely describe its role. Strictly speaking, the name isn't perfect as toggling virtual VM{LOAD,SAVE} is arguably not recalculating an intercept, but practically speaking it's close enough.
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/svm.c +4 -3
@@ -1107,7 +1107,7 @@ static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu, } } -static inline void init_vmcb_after_set_cpuid (struct kvm_vcpu *vcpu) +static void svm_recalc_intercepts_after_set_cpuid (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -1273,7 +1273,8 @@ static void init_vmcb(struct kvm_vcpu *vcpu)sev_init_vmcb (svm);svm_hv_init_vmcb (vmcb); -init_vmcb_after_set_cpuid (vcpu); + +svm_recalc_intercepts_after_set_cpuid (vcpu);vmcb_mark_all_dirty (vmcb);
@@ -4522,7 +4523,7 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (sev_guest (vcpu->kvm))sev_vcpu_after_set_cpuid (svm); -init_vmcb_after_set_cpuid (vcpu); +svm_recalc_intercepts_after_set_cpuid (vcpu); }static bool svm_has_wbinvd_exit (void )
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 20/28 KVM: SVM: Fold svm_vcpu_init_msrpm() into its sole caller
In reply to Sean ChristophersonFold svm_vcpu_init_msrpm() into svm_recalc_msr_intercepts() now that there is only the one caller (and because the "init" misnomer is even more misleading than it was in the past).
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-)
arch/x86/kvm/svm/svm.c +11 -16
@@ -787,21 +787,6 @@ static void svm_recalc_lbr_msr_intercepts(struct kvm_vcpu *vcpu) } -static void svm_vcpu_init_msrpm (struct kvm_vcpu *vcpu) -{ -svm_disable_intercept_for_msr (vcpu,MSR_STAR ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); - -#ifdef CONFIG_X86_64 -svm_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_LSTAR ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_CSTAR ,MSR_TYPE_RW ); -svm_disable_intercept_for_msr (vcpu,MSR_SYSCALL_MASK ,MSR_TYPE_RW ); -#endif -} -void svm_set_x2apic_msr_interception (struct vcpu_svm *svm,bool intercept) {static const u32 x2avic_passthrough_msrs[] = {
@@ -860,7 +845,17 @@ static void svm_recalc_msr_intercepts(struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); -svm_vcpu_init_msrpm (vcpu); +svm_disable_intercept_for_msr (vcpu,MSR_STAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_IA32_SYSENTER_CS ,MSR_TYPE_RW ); + +#ifdef CONFIG_X86_64 +svm_disable_intercept_for_msr (vcpu,MSR_GS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_FS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_KERNEL_GS_BASE ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_LSTAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_CSTAR ,MSR_TYPE_RW ); +svm_disable_intercept_for_msr (vcpu,MSR_SYSCALL_MASK ,MSR_TYPE_RW ); +#endif if (lbrv)svm_recalc_lbr_msr_intercepts (vcpu);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 21/28 KVM: SVM: Merge "after set CPUID" intercept recalc helpers
In reply to Sean ChristophersonMerge svm_recalc_intercepts_after_set_cpuid() and svm_recalc_instruction_intercepts() such that the "after set CPUID" helper simply invokes the type-specific helpers (MSRs vs. instructions), i.e. make svm_recalc_intercepts_after_set_cpuid() a single entry point for all intercept updates that need to be performed after a CPUID change.
No functional change intended.
--- arch/x86/kvm/svm/svm.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-)
arch/x86/kvm/svm/svm.c +7 -11
@@ -1079,9 +1079,10 @@ void svm_write_tsc_multiplier(struct kvm_vcpu *vcpu) }/* Evaluate instruction intercepts that depend on guest CPUID features. */ -static void svm_recalc_instruction_intercepts (struct kvm_vcpu *vcpu, -struct vcpu_svm *svm) +static void svm_recalc_instruction_intercepts (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu); +/* * Intercept INVPCID if shadow paging is enabled to sync/free shadow * roots, or if INVPCID is disabled in the guest to inject #UD.
@@ -1100,11 +1101,6 @@ static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu,else svm_set_intercept (svm,INTERCEPT_RDTSCP ); } -} - -static void svm_recalc_intercepts_after_set_cpuid (struct kvm_vcpu *vcpu) -{ -struct vcpu_svm *svm =to_svm (vcpu);if (guest_cpuid_is_intel_compatible (vcpu)) {svm_set_intercept (svm,INTERCEPT_VMLOAD );
@@ -1121,7 +1117,11 @@ static void svm_recalc_intercepts_after_set_cpuid(struct kvm_vcpu *vcpu) svm->vmcb->control.virt_ext |=VIRTUAL_VMLOAD_VMSAVE_ENABLE_MASK ; } } +} +static void svm_recalc_intercepts_after_set_cpuid (struct kvm_vcpu *vcpu) +{ +svm_recalc_instruction_intercepts (vcpu);svm_recalc_msr_intercepts (vcpu); }
@@ -1247,8 +1247,6 @@ static void init_vmcb(struct kvm_vcpu *vcpu)svm_clr_intercept (svm,INTERCEPT_PAUSE ); } -svm_recalc_instruction_intercepts (vcpu, svm); -if (kvm_vcpu_apicv_active (vcpu))avic_init_vmcb (svm, vmcb);
@@ -4513,8 +4511,6 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)if (guest_cpuid_is_intel_compatible (vcpu))guest_cpu_cap_clear (vcpu,X86_FEATURE_V_VMSAVE_VMLOAD ); -svm_recalc_instruction_intercepts (vcpu, svm); -if (sev_guest (vcpu->kvm))sev_vcpu_after_set_cpuid (svm);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 22/28 KVM: SVM: Drop explicit check on MSRPM offset when emulating SEV-ES accesses
In reply to Sean ChristophersonNow that msr_write_intercepted() defaults to true, i.e. accurately reflects hardware behavior for out-of-range MSRs, and doesn't WARN (or BUG) on an out-of-range MSR, drop sev_es_prevent_msr_access()'s svm_msrpm_offset() check that guarded against calling msr_write_intercepted() with a "bad" index.
Opportunistically clean up the helper's formatting.
--- arch/x86/kvm/svm/svm.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/svm/svm.c +2 -3
@@ -2771,12 +2771,11 @@ static int svm_get_feature_msr(u32 msr, u64 *data)return 0 ; } -static bool -sev_es_prevent_msr_access (struct kvm_vcpu *vcpu,struct msr_data *msr_info) +static bool sev_es_prevent_msr_access (struct kvm_vcpu *vcpu, +struct msr_data *msr_info) {return sev_es_guest (vcpu->kvm) && vcpu->arch.guest_state_protected && -svm_msrpm_offset (msr_info->index) !=MSR_INVALID && !msr_write_intercepted (vcpu, msr_info->index); }
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 23/28 KVM: SVM: Move svm_msrpm_offset() to nested.c
In reply to Sean ChristophersonMove svm_msrpm_offset() from svm.c to nested.c now that all usage of the u32-index offsets is nested virtualization specific.
No functional change intended.
--- arch/x86/kvm/svm/nested.c | 27 +++++++++++++++++++++++++++ arch/x86/kvm/svm/svm.c | 27 --------------------------- arch/x86/kvm/svm/svm.h | 1 - 3 files changed, 27 insertions(+), 28 deletions(-)
arch/x86/kvm/svm/nested.c +27 -0
@@ -187,6 +187,33 @@ void recalc_intercepts(struct vcpu_svm *svm)static int nested_svm_msrpm_merge_offsets[6 ]__ro_after_init ;static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; +static const u32 msrpm_ranges[] = { +SVM_MSRPM_RANGE_0_BASE_MSR , +SVM_MSRPM_RANGE_1_BASE_MSR , +SVM_MSRPM_RANGE_2_BASE_MSR +}; + +static u32 svm_msrpm_offset (u32 msr) +{ +u32 offset; +int i; + +for (i =0 ; i <ARRAY_SIZE (msrpm_ranges); i++) { +if (msr < msrpm_ranges[i] || + msr >= msrpm_ranges[i] +SVM_MSRS_PER_RANGE ) +continue ; + + offset = (msr - msrpm_ranges[i]) /SVM_MSRS_PER_BYTE ; + offset += (i *SVM_MSRPM_BYTES_PER_RANGE );/* add range offset */ + +/* Now we have the u8 offset - but need the u32 offset */ +return offset /4 ; + } + +/* MSR not in any range */ +return MSR_INVALID ; +} +int __init nested_svm_init_msrpm_merge_offsets (void ) {const u32 merge_msrs[] = {
arch/x86/kvm/svm/svm.c +0 -27
@@ -195,33 +195,6 @@ static DEFINE_MUTEX(vmcb_dump_mutex);*/ static int tsc_aux_uret_slot__read_mostly =-1 ; -static const u32 msrpm_ranges[] = { -SVM_MSRPM_RANGE_0_BASE_MSR , -SVM_MSRPM_RANGE_1_BASE_MSR , -SVM_MSRPM_RANGE_2_BASE_MSR -}; - -u32 svm_msrpm_offset (u32 msr) -{ -u32 offset; -int i; - -for (i =0 ; i <ARRAY_SIZE (msrpm_ranges); i++) { -if (msr < msrpm_ranges[i] || - msr >= msrpm_ranges[i] +SVM_MSRS_PER_RANGE ) -continue ; - - offset = (msr - msrpm_ranges[i]) /SVM_MSRS_PER_BYTE ; - offset += (i *SVM_MSRPM_BYTES_PER_RANGE );/* add range offset */ - -/* Now we have the u8 offset - but need the u32 offset */ -return offset /4 ; - } - -/* MSR not in any range */ -return MSR_INVALID ; -} -static int get_npt_level (void ) {#ifdef CONFIG_X86_64
arch/x86/kvm/svm/svm.h +0 -1
@@ -673,7 +673,6 @@ BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set)/* svm.c */ extern bool dump_invalid_vmcb; -u32 svm_msrpm_offset (u32 msr);u32 *svm_vcpu_alloc_msrpm (void );void svm_vcpu_free_msrpm (u32 *msrpm);void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 24/28 KVM: SVM: Store MSRPM pointer as "void *" instead of "u32 *"
In reply to Sean ChristophersonStore KVM's MSRPM pointers as "void *" instead of "u32 *" to guard against directly accessing the bitmaps outside of code that is explicitly written to access the bitmaps with a specific type.
Opportunistically use svm_vcpu_free_msrpm() in svm_vcpu_free() instead of open coding an equivalent.
--- arch/x86/kvm/svm/nested.c | 4 +++- arch/x86/kvm/svm/svm.c | 8 ++++---- arch/x86/kvm/svm/svm.h | 13 ++++++++----- 3 files changed, 15 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +3 -1
@@ -271,6 +271,8 @@ int __init nested_svm_init_msrpm_merge_offsets(void)static bool nested_svm_merge_msrpm (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); +u32 *msrpm02 = svm->nested.msrpm; +u32 *msrpm01 = svm->msrpm;int i;/*
@@ -305,7 +307,7 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)if (kvm_vcpu_read_guest (vcpu, offset, &value,4 ))return false; - svm->nested.msrpm[p] = svm->msrpm[p] | value; + msrpm02[p] = msrpm01[p] | value; } svm->nested.force_msr_bitmap_recalc = false;
arch/x86/kvm/svm/svm.c +4 -4
@@ -731,11 +731,11 @@ void svm_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type) svm->nested.force_msr_bitmap_recalc = true; } -u32 *svm_vcpu_alloc_msrpm (void ) +void *svm_vcpu_alloc_msrpm (void ) {unsigned int order =get_order (MSRPM_SIZE );struct page *pages =alloc_pages (GFP_KERNEL_ACCOUNT , order); -u32 *msrpm; +void *msrpm;if (!pages)return NULL ;
@@ -809,7 +809,7 @@ void svm_set_x2apic_msr_interception(struct vcpu_svm *svm, bool intercept) svm->x2avic_msrs_intercepted = intercept; } -void svm_vcpu_free_msrpm (u32 *msrpm) +void svm_vcpu_free_msrpm (void *msrpm) {__free_pages (virt_to_page (msrpm),get_order (MSRPM_SIZE )); }
@@ -1353,7 +1353,7 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu)sev_free_vcpu (vcpu);__free_page (__sme_pa_to_page (svm->vmcb01.pa)); -__free_pages (virt_to_page (svm->msrpm),get_order (MSRPM_SIZE )); +svm_vcpu_free_msrpm (svm->msrpm); }#ifdef CONFIG_CPU_MITIGATIONS
arch/x86/kvm/svm/svm.h +8 -5
@@ -189,8 +189,11 @@ struct svm_nested_state {u64 vmcb12_gpa;u64 last_vmcb12_gpa; -/* These are the merged vectors */ -u32 *msrpm; +/* +* The MSR permissions map used for vmcb02, which is the merge result +* of vmcb01 and vmcb12 +*/ +void *msrpm;/* A VMRUN has started but has not yet been performed, so * we cannot inject a nested vmexit yet. */
@@ -271,7 +274,7 @@ struct vcpu_svm {*/ u64 virt_spec_ctrl; -u32 *msrpm; +void *msrpm;ulong nmi_iret_rip;
@@ -673,8 +676,8 @@ BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set)/* svm.c */ extern bool dump_invalid_vmcb; -u32 *svm_vcpu_alloc_msrpm (void ); -void svm_vcpu_free_msrpm (u32 *msrpm); +void *svm_vcpu_alloc_msrpm (void ); +void svm_vcpu_free_msrpm (void *msrpm);void svm_copy_lbrs (struct vmcb *to_vmcb,struct vmcb *from_vmcb);void svm_enable_lbrv (struct kvm_vcpu *vcpu);void svm_update_lbrv (struct kvm_vcpu *vcpu);
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 25/28 KVM: nSVM: Access MSRPM in 4-byte chunks only for merging L0 and L1 bitmaps
In reply to Sean ChristophersonAccess the MSRPM using u32/4-byte chunks (and appropriately adjusted offsets) only when merging L0 and L1 bitmaps as part of emulating VMRUN. The only reason to batch accesses to MSRPMs is to avoid the overhead of uaccess operations (e.g. STAC/CLAC and bounds checks) when reading L1's bitmap pointed at by vmcb12. For all other uses, either per-bit accesses are more than fast enough (no uaccess), or KVM is only accessing a single bit (nested_svm_exit_handled_msr()) and so there's nothing to batch.
In addition to (hopefully) documenting the uniqueness of the merging code, restricting chunked access to _just_ the merging code will allow for increasing the chunk size (to unsigned long) with minimal risk.
--- arch/x86/kvm/svm/nested.c | 50 ++++++++++++++++----------------------- arch/x86/kvm/svm/svm.h | 18 ++++++++++---- 2 files changed, 34 insertions(+), 34 deletions(-)
arch/x86/kvm/svm/nested.c +21 -29
@@ -187,31 +187,19 @@ void recalc_intercepts(struct vcpu_svm *svm)static int nested_svm_msrpm_merge_offsets[6 ]__ro_after_init ;static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; -static const u32 msrpm_ranges[] = { -SVM_MSRPM_RANGE_0_BASE_MSR , -SVM_MSRPM_RANGE_1_BASE_MSR , -SVM_MSRPM_RANGE_2_BASE_MSR -}; +#define SVM_BUILD_MSR_BYTE_NR_CASE (range_nr, msr) \ + case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \ + return SVM_MSRPM_BYTE_NR(range_nr, msr);static u32 svm_msrpm_offset (u32 msr) { -u32 offset; -int i; - -for (i =0 ; i <ARRAY_SIZE (msrpm_ranges); i++) { -if (msr < msrpm_ranges[i] || - msr >= msrpm_ranges[i] +SVM_MSRS_PER_RANGE ) -continue ; - - offset = (msr - msrpm_ranges[i]) /SVM_MSRS_PER_BYTE ; - offset += (i *SVM_MSRPM_BYTES_PER_RANGE );/* add range offset */ - -/* Now we have the u8 offset - but need the u32 offset */ -return offset /4 ; +switch (msr) { + SVM_BUILD_MSR_BYTE_NR_CASE(0, msr) + SVM_BUILD_MSR_BYTE_NR_CASE(1, msr) + SVM_BUILD_MSR_BYTE_NR_CASE(2, msr) +default : +return MSR_INVALID ; } - -/* MSR not in any range */ -return MSR_INVALID ; }int __init nested_svm_init_msrpm_merge_offsets (void )
@@ -245,6 +233,12 @@ int __init nested_svm_init_msrpm_merge_offsets(void)if (WARN_ON (offset ==MSR_INVALID ))return -EIO ; +/* +* Merging is done in 32-bit chunks to reduce the number of +* accesses to L1's bitmap. +*/ + offset /=sizeof (u32); +for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) {if (nested_svm_msrpm_merge_offsets[j] == offset)break ;
@@ -1363,8 +1357,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)static int nested_svm_exit_handled_msr (struct vcpu_svm *svm) { -u32 offset, msr, value; -int write, mask; +u32 offset, msr; +int write; +u8 value;if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return NESTED_EXIT_HOST ;
@@ -1372,18 +1367,15 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) msr = svm->vcpu.arch.regs[VCPU_REGS_RCX ]; offset =svm_msrpm_offset (msr); write = svm->vmcb->control.exit_info_1 &1 ; - mask =1 << ((2 * (msr &0xf )) + write);if (offset ==MSR_INVALID )return NESTED_EXIT_DONE ; -/* Offset is in 32 bit units but need in 8 bit units */ - offset *=4 ; - -if (kvm_vcpu_read_guest (&svm->vcpu, svm->nested.ctl.msrpm_base_pa + offset, &value,4 )) +if (kvm_vcpu_read_guest (&svm->vcpu, svm->nested.ctl.msrpm_base_pa + offset, + &value,sizeof (value)))return NESTED_EXIT_DONE ; -return (value & mask) ?NESTED_EXIT_DONE :NESTED_EXIT_HOST ; +return (value &BIT (write)) ?NESTED_EXIT_DONE :NESTED_EXIT_HOST ; }static int nested_svm_intercept_ioio (struct vcpu_svm *svm)
arch/x86/kvm/svm/svm.h +13 -5
@@ -634,15 +634,23 @@ static_assert(SVM_MSRS_PER_RANGE == 8192); (range_nr *SVM_MSRPM_BYTES_PER_RANGE *BITS_PER_BYTE + \ (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) * SVM_BITS_PER_MSR) -#define SVM_MSRPM_SANITY_CHECK_BITS (range_nr) \ +#define SVM_MSRPM_BYTE_NR (range_nr, msr) \ + (range_nr * SVM_MSRPM_BYTES_PER_RANGE + \ + (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) / SVM_MSRS_PER_BYTE) + +#define SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (range_nr) \ static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ range_nr * 2048 * 8 + 2); \ static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ - range_nr * 2048 * 8 + 14); + range_nr * 2048 * 8 + 14); \ +static_assert(SVM_MSRPM_BYTE_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ + range_nr * 2048); \ +static_assert(SVM_MSRPM_BYTE_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ + range_nr * 2048 + 1); -SVM_MSRPM_SANITY_CHECK_BITS (0 ); -SVM_MSRPM_SANITY_CHECK_BITS (1 ); -SVM_MSRPM_SANITY_CHECK_BITS (2 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (0 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (1 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (2 );#define SVM_BUILD_MSR_BITMAP_CASE (bitmap, range_nr, msr, bitop, bit_rw) \ case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Paolo Bonzini
maintainerRe: [PATCH 25/28] KVM: nSVM: Access MSRPM in 4-byte chunks only for merging L0 and L1 bitmaps
Quoted 89 lines · Sean Christopherson
Access the MSRPM using u32/4-byte chunks (and appropriately adjusted offsets) only when merging L0 and L1 bitmaps as part of emulating VMRUN. The only reason to batch accesses to MSRPMs is to avoid the overhead of uaccess operations (e.g. STAC/CLAC and bounds checks) when reading L1's bitmap pointed at by vmcb12. For all other uses, either per-bit accesses are more than fast enough (no uaccess), or KVM is only accessing a single bit (
nested_svm_exit_handled_msr()) and so there's nothing to batch.In addition to (hopefully) documenting the uniqueness of the merging code, restricting chunked access to _just_ the merging code will allow for increasing the chunk size (to unsigned long) with minimal risk.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/svm/nested.c | 50 ++++++++++++++++----------------------- arch/x86/kvm/svm/svm.h | 18 ++++++++++---- 2 files changed, 34 insertions(+), 34 deletions(-)arch/x86/kvm/svm/nested.c +18
-23@@ -187,31 +187,19 @@ void recalc_intercepts(struct vcpu_svm *svm)static int nested_svm_msrpm_merge_offsets[6 ]__ro_after_init ;static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ; -static const u32 msrpm_ranges[] = { -SVM_MSRPM_RANGE_0_BASE_MSR , -SVM_MSRPM_RANGE_1_BASE_MSR , -SVM_MSRPM_RANGE_2_BASE_MSR -}; +#define SVM_BUILD_MSR_BYTE_NR_CASE (range_nr, msr) \ + case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \ + return SVM_MSRPM_BYTE_NR(range_nr, msr);static u32 svm_msrpm_offset (u32 msr) { -u32 offset; -int i; - -for (i =0 ; i <ARRAY_SIZE (msrpm_ranges); i++) { -if (msr < msrpm_ranges[i] || - msr >= msrpm_ranges[i] +SVM_MSRS_PER_RANGE ) -continue ; - - offset = (msr - msrpm_ranges[i]) /SVM_MSRS_PER_BYTE ; - offset += (i *SVM_MSRPM_BYTES_PER_RANGE );/* add range offset */ - -/* Now we have the u8 offset - but need the u32 offset */ -return offset /4 ; +switch (msr) { + SVM_BUILD_MSR_BYTE_NR_CASE(0, msr) + SVM_BUILD_MSR_BYTE_NR_CASE(1, msr) + SVM_BUILD_MSR_BYTE_NR_CASE(2, msr) +default : +return MSR_INVALID ; } - -/* MSR not in any range */ -return MSR_INVALID ; }int __init nested_svm_init_msrpm_merge_offsets (void )@@ -245,6 +233,12 @@ int __init nested_svm_init_msrpm_merge_offsets(void)if (WARN_ON (offset ==MSR_INVALID ))return -EIO ; +/* +* Merging is done in 32-bit chunks to reduce the number of +* accesses to L1's bitmap. +*/ + offset /=sizeof (u32); +for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) {if (nested_svm_msrpm_merge_offsets[j] == offset)break ;@@ -1363,8 +1357,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)static int nested_svm_exit_handled_msr (struct vcpu_svm *svm) { -u32 offset, msr, value; -int write, mask; +u32 offset, msr; +int write; +u8 value;if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return NESTED_EXIT_HOST ;
arch/x86/kvm/svm/nested.c static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) · line 1375
Quoted 4 lines
arch/x86/kvm/svm/nested.c +0
-1@@ -1372,18 +1367,15 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) msr = svm->vcpu.arch.regs[VCPU_REGS_RCX ]; offset =svm_msrpm_offset (msr); write = svm->vmcb->control.exit_info_1 &1 ; - mask =1 << ((2 * (msr &0xf )) + write);
This is wrong. The bit to read isn't always bit 0 or bit 1, therefore mask needs to remain. But it can be written easily as:
msr = svm->vcpu.arch.regs[VCPU_REGS_RCX]; write = svm->vmcb->control.exit_info_1 & 1; bit = svm_msrpm_bit(msr); if (bit == MSR_INVALID) return NESTED_EXIT_DONE; offset = bit / BITS_PER_BYTE; mask = BIT(write) << (bit & (BITS_PER_BYTE - 1));
and this even removes the need to use svm_msrpm_offset() in nested_svm_exit_handled_msr().
At this point, it may even make sense to keep the adjustment for the offset in svm_msrpm_offset(), like this:
static u32 svm_msrpm_offset(u32 msr)
{
u32 bit = svm_msr_bit(msr);
if (bit == MSR_INVALID)
return MSR_INVALID;
/*
* Merging is done in 32-bit chunks to reduce the number of
* accesses to L1's bitmap.
*/
return bit / (BITS_PER_BYTE * sizeof(u32));
}I'll let you be the judge on this.
Paolo
Quoted 49 lines
arch/x86/kvm/svm/nested.c +3
-5if (offset ==MSR_INVALID )return NESTED_EXIT_DONE ; -/* Offset is in 32 bit units but need in 8 bit units */ - offset *=4 ; - -if (kvm_vcpu_read_guest (&svm->vcpu, svm->nested.ctl.msrpm_base_pa + offset, &value,4 )) +if (kvm_vcpu_read_guest (&svm->vcpu, svm->nested.ctl.msrpm_base_pa + offset, + &value,sizeof (value)))return NESTED_EXIT_DONE ; -return (value & mask) ?NESTED_EXIT_DONE :NESTED_EXIT_HOST ; +return (value &BIT (write)) ?NESTED_EXIT_DONE :NESTED_EXIT_HOST ; }static int nested_svm_intercept_ioio (struct vcpu_svm *svm)arch/x86/kvm/svm/svm.h +13
-5@@ -634,15 +634,23 @@ static_assert(SVM_MSRS_PER_RANGE == 8192); (range_nr *SVM_MSRPM_BYTES_PER_RANGE *BITS_PER_BYTE + \ (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) * SVM_BITS_PER_MSR) -#define SVM_MSRPM_SANITY_CHECK_BITS (range_nr) \ +#define SVM_MSRPM_BYTE_NR (range_nr, msr) \ + (range_nr * SVM_MSRPM_BYTES_PER_RANGE + \ + (msr - SVM_MSRPM_RANGE_## range_nr ##_BASE_MSR) / SVM_MSRS_PER_BYTE) + +#define SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (range_nr) \ static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ range_nr * 2048 * 8 + 2); \ static_assert(SVM_MSRPM_BIT_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ - range_nr * 2048 * 8 + 14); + range_nr * 2048 * 8 + 14); \ +static_assert(SVM_MSRPM_BYTE_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 1) == \ + range_nr * 2048); \ +static_assert(SVM_MSRPM_BYTE_NR(range_nr, SVM_MSRPM_FIRST_MSR(range_nr) + 7) == \ + range_nr * 2048 + 1); -SVM_MSRPM_SANITY_CHECK_BITS (0 ); -SVM_MSRPM_SANITY_CHECK_BITS (1 ); -SVM_MSRPM_SANITY_CHECK_BITS (2 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (0 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (1 ); +SVM_MSRPM_SANITY_CHECK_BITS_AND_BYTES (2 );#define SVM_BUILD_MSR_BITMAP_CASE (bitmap, range_nr, msr, bitop, bit_rw) \ case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \
Sean Christopherson
maintainerRe: [PATCH 25/28] KVM: nSVM: Access MSRPM in 4-byte chunks only for merging L0 and L1 bitmaps
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Sean Christopherson
arch/x86/kvm/svm/nested.c +3
-2@@ -1363,8 +1357,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)static int nested_svm_exit_handled_msr (struct vcpu_svm *svm) { -u32 offset, msr, value; -int write, mask; +u32 offset, msr; +int write; +u8 value;if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return NESTED_EXIT_HOST ;
Quoted 8 lines
arch/x86/kvm/svm/nested.c static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) · line 1375
Quoted 5 lines
arch/x86/kvm/svm/nested.c +0
-1@@ -1372,18 +1367,15 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) msr = svm->vcpu.arch.regs[VCPU_REGS_RCX ]; offset =svm_msrpm_offset (msr); write = svm->vmcb->control.exit_info_1 &1 ; - mask =1 << ((2 * (msr &0xf )) + write);This is wrong. The bit to read isn't always bit 0 or bit 1, therefore mask needs to remain.
/facepalm
Duh. I managed to forget that multiple MSRs are packed into a byte. Hrm, which means our nSVM test is even more worthless than I thought. I'll see if I can get it to detect this bug.
Sean Christopherson
maintainerPATCH 26/28 KVM: SVM: Return -EINVAL instead of MSR_INVALID to signal out-of-range MSR
In reply to Sean ChristophersonReturn -EINVAL instead of MSR_INVALID from svm_msrpm_offset() to indicate that the MSR isn't covered by one of the (currently) three MSRPM ranges, and delete the MSR_INVALID macro now that all users are gone.
--- arch/x86/kvm/svm/nested.c | 14 +++++++------- arch/x86/kvm/svm/svm.h | 2 -- 2 files changed, 7 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/nested.c +7 -7
@@ -191,14 +191,14 @@ static int nested_svm_nr_msrpm_merge_offsets __ro_after_init; case SVM_MSRPM_FIRST_MSR(range_nr) ... SVM_MSRPM_LAST_MSR(range_nr): \return SVM_MSRPM_BYTE_NR (range_nr, msr); -static u32 svm_msrpm_offset(u32 msr) +static int svm_msrpm_offset(u32 msr) {switch (msr) { SVM_BUILD_MSR_BYTE_NR_CASE(0, msr) SVM_BUILD_MSR_BYTE_NR_CASE(1, msr) SVM_BUILD_MSR_BYTE_NR_CASE(2, msr)default : -return MSR_INVALID ; +return -EINVAL ; } }
@@ -228,9 +228,9 @@ int __init nested_svm_init_msrpm_merge_offsets(void)int i, j;for (i =0 ; i <ARRAY_SIZE (merge_msrs); i++) { -u32 offset =svm_msrpm_offset (merge_msrs[i]); +int offset =svm_msrpm_offset (merge_msrs[i]); -if (WARN_ON (offset ==MSR_INVALID )) +if (WARN_ON (offset <0 ))return -EIO ;/*
@@ -1357,9 +1357,9 @@ void svm_leave_nested(struct kvm_vcpu *vcpu)static int nested_svm_exit_handled_msr (struct vcpu_svm *svm) { -u32 offset, msr; -int write; +int offset, write;u8 value; +u32 msr;if (!(vmcb12_is_intercept (&svm->nested.ctl,INTERCEPT_MSR_PROT )))return NESTED_EXIT_HOST ;
@@ -1368,7 +1368,7 @@ static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) offset =svm_msrpm_offset (msr); write = svm->vmcb->control.exit_info_1 &1 ; -if (offset ==MSR_INVALID ) +if (offset <0 )return NESTED_EXIT_DONE ; if (kvm_vcpu_read_guest(&svm->vcpu, svm->nested.ctl.msrpm_base_pa + offset,
arch/x86/kvm/svm/svm.h +0 -2
@@ -677,8 +677,6 @@ BUILD_SVM_MSR_BITMAP_HELPERS(bool, test, test) BUILD_SVM_MSR_BITMAP_HELPERS(void, clear, __clear) BUILD_SVM_MSR_BITMAP_HELPERS(void, set, __set) -#define MSR_INVALID 0xffffffffU -#define DEBUGCTL_RESERVED_BITS (~DEBUGCTLMSR_LBR)/* svm.c */
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 27/28 KVM: nSVM: Merge MSRPM in 64-bit chunks on 64-bit kernels
In reply to Sean ChristophersonWhen merging L0 and L1 MSRPMs as part of nested VMRUN emulation, access the bitmaps using "unsigned long" chunks, i.e. use 8-byte access for 64-bit kernels instead of arbitrarily working on 4-byte chunks.
Opportunistically rename local variables in nested_svm_merge_msrpm() to more precisely/accurately reflect their purpose ("offset" in particular is extremely ambiguous).
--- arch/x86/kvm/svm/nested.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/nested.c +11 -10
@@ -184,6 +184,7 @@ void recalc_intercepts(struct vcpu_svm *svm) } } +typedef unsigned long nsvm_msrpm_merge_t ;static int nested_svm_msrpm_merge_offsets[6 ]__ro_after_init ;static int nested_svm_nr_msrpm_merge_offsets__ro_after_init ;
@@ -234,10 +235,10 @@ int __init nested_svm_init_msrpm_merge_offsets(void)return -EIO ;/* -* Merging is done in 32-bit chunks to reduce the number of -* accesses to L1's bitmap. +* Merging is done in chunks to reduce the number of accesses +* to L1's bitmap. */ - offset /=sizeof (u32); + offset /=sizeof (nsvm_msrpm_merge_t);for (j =0 ; j < nested_svm_nr_msrpm_merge_offsets; j++) {if (nested_svm_msrpm_merge_offsets[j] == offset)
@@ -265,8 +266,8 @@ int __init nested_svm_init_msrpm_merge_offsets(void)static bool nested_svm_merge_msrpm (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu); -u32 *msrpm02 = svm->nested.msrpm; -u32 *msrpm01 = svm->msrpm; +nsvm_msrpm_merge_t *msrpm02 = svm->nested.msrpm; +nsvm_msrpm_merge_t *msrpm01 = svm->msrpm;int i;/*
@@ -293,15 +294,15 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu)for (i =0 ; i < nested_svm_nr_msrpm_merge_offsets; i++) {const int p = nested_svm_msrpm_merge_offsets[i]; -u32 value; -u64 offset; +nsvm_msrpm_merge_t l1_val; +gpa_t gpa; - offset = svm->nested.ctl.msrpm_base_pa + (p *4 ); + gpa = svm->nested.ctl.msrpm_base_pa + (p *sizeof (l1_val)); -if (kvm_vcpu_read_guest (vcpu, offset, &value,4 )) +if (kvm_vcpu_read_guest (vcpu, gpa, &l1_val,sizeof (l1_val)))return false; - msrpm02[p] = msrpm01[p] | value; + msrpm02[p] = msrpm01[p] | l1_val; } svm->nested.force_msr_bitmap_recalc = false;
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Sean Christopherson
maintainerPATCH 28/28 KVM: selftests: Verify KVM disable interception (for userspace) on filter change
In reply to Sean ChristophersonRe-read MSR_{FS,GS}_BASE after restoring the "allow everything" userspace MSR filter to verify that KVM stops forwarding exits to userspace. This can also be used in conjunction with manual verification (e.g. printk) to ensure KVM is correctly updating the MSR bitmaps consumed by hardware.
--- tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c | 8 ++++++++ 1 file changed, 8 insertions(+)
tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c +8 -0
@@ -343,6 +343,12 @@ static void guest_code_permission_bitmap(void) data =test_rdmsr (MSR_GS_BASE );GUEST_ASSERT (data ==MSR_GS_BASE ); +/* Access the MSRs again to ensure KVM has disabled interception.*/ + data =test_rdmsr (MSR_FS_BASE ); +GUEST_ASSERT (data !=MSR_FS_BASE ); + data =test_rdmsr (MSR_GS_BASE ); +GUEST_ASSERT (data !=MSR_GS_BASE ); +GUEST_DONE (); }
@@ -682,6 +688,8 @@ KVM_ONE_VCPU_TEST(user_msr, msr_permission_bitmap, guest_code_permission_bitmap) "Expected ucall state to be UCALL_SYNC.");vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_gs);run_guest_then_process_rdmsr (vcpu,MSR_GS_BASE ); + +vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_allow);run_guest_then_process_ucall_done (vcpu); }
Signature · 1 line
2.49.0.1204.g71687c7c1d-goog
Re: [PATCH 28/28] KVM: selftests: Verify KVM disable interception (for userspace) on filter change
Quoted 28 lines · Sean Christopherson
Re-read MSR_{FS,GS}_BASE after restoring the "allow everything" userspace MSR filter to verify that KVM stops forwarding exits to userspace. This can also be used in conjunction with manual verification (e.g. printk) to ensure KVM is correctly updating the MSR bitmaps consumed by hardware.
Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c | 8 ++++++++ 1 file changed, 8 insertions(+)tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c +6
-0@@ -343,6 +343,12 @@ static void guest_code_permission_bitmap(void) data =test_rdmsr (MSR_GS_BASE );GUEST_ASSERT (data ==MSR_GS_BASE ); +/* Access the MSRs again to ensure KVM has disabled interception.*/ + data =test_rdmsr (MSR_FS_BASE ); +GUEST_ASSERT (data !=MSR_FS_BASE ); + data =test_rdmsr (MSR_GS_BASE ); +GUEST_ASSERT (data !=MSR_GS_BASE ); +GUEST_DONE (); }
tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c KVM_ONE_VCPU_TEST(user_msr, msr_permission_bitmap, guest_code_permission_bitmap) · line 695
Quoted 8 lines
tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c +2
-0@@ -682,6 +688,8 @@ KVM_ONE_VCPU_TEST(user_msr, msr_permission_bitmap, guest_code_permission_bitmap) "Expected ucall state to be UCALL_SYNC.");vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_gs);run_guest_then_process_rdmsr (vcpu,MSR_GS_BASE ); + +vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_allow);run_guest_then_process_ucall_done (vcpu); }
Test passes on Intel platform (Sapphire Rapids).
Manali Shukla
Tested-byRe: [PATCH 28/28] KVM: selftests: Verify KVM disable interception (for userspace) on filter change
Quoted 41 lines · Dapeng
Quoted 28 lines · Sean Christopherson
Re-read MSR_{FS,GS}_BASE after restoring the "allow everything" userspace MSR filter to verify that KVM stops forwarding exits to userspace. This can also be used in conjunction with manual verification (e.g. printk) to ensure KVM is correctly updating the MSR bitmaps consumed by hardware.
Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c | 8 ++++++++ 1 file changed, 8 insertions(+)tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c +6
-0@@ -343,6 +343,12 @@ static void guest_code_permission_bitmap(void) data =test_rdmsr (MSR_GS_BASE );GUEST_ASSERT (data ==MSR_GS_BASE ); +/* Access the MSRs again to ensure KVM has disabled interception.*/ + data =test_rdmsr (MSR_FS_BASE ); +GUEST_ASSERT (data !=MSR_FS_BASE ); + data =test_rdmsr (MSR_GS_BASE ); +GUEST_ASSERT (data !=MSR_GS_BASE ); +GUEST_DONE (); }tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c KVM_ONE_VCPU_TEST(user_msr, msr_permission_bitmap, guest_code_permission_bitmap) · line 695
Quoted 8 lines
tools/testing/selftests/kvm/x86/userspace_msr_exit_test.c +2
-0@@ -682,6 +688,8 @@ KVM_ONE_VCPU_TEST(user_msr, msr_permission_bitmap, guest_code_permission_bitmap) "Expected ucall state to be UCALL_SYNC.");vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_gs);run_guest_then_process_rdmsr (vcpu,MSR_GS_BASE ); + +vm_ioctl (vm,KVM_X86_SET_MSR_FILTER , &filter_allow);run_guest_then_process_ucall_done (vcpu); }Test passes on Intel platform (Sapphire Rapids).
Quoted 3 lines
Tested-by Dapeng Mi <[email protected]>
This test passes on AMD platform (Genoa)