KVM: nVMX: Improve performance for unmanaged guest memory
v4 cover KVM: nVMX: Improve performance for unmanaged guest memory
From: Fred Griffoul <[email protected]>
This patch series addresses both performance and correctness issues in nested VMX when handling guest memory.
During nested VMX operations, L0 (KVM) accesses specific L1 guest pages to manage L2 execution. These pages fall into two categories: pages accessed only by L0 (such as the L1 MSR bitmap page or the eVMCS page), and pages passed to the L2 guest via vmcs02 (such as APIC access, virtual APIC, and posted interrupt descriptor pages).
The current implementation uses kvm_vcpu_map/unmap, which causes two issues.
First, the current approach is missing proper invalidation handling in critical scenarios. Enlightened VMCS (eVMCS) pages can become stale when memslots are modified, as there is no mechanism to invalidate the cached mappings. Similarly, APIC access and virtual APIC pages can be migrated by the host, but without proper notification through mmu_notifier callbacks, the mappings become invalid and can lead to incorrect behavior.
Second, for unmanaged guest memory (memory not directly mapped by the kernel, such as memory passed with the mem= parameter or guest_memfd for non-CoCo VMs), this workflow invokes expensive memremap/memunmap operations on every L2 VM entry/exit cycle. This creates significant overhead that impacts nested virtualization performance.
This series replaces kvm_host_map with gfn_to_pfn_cache in nested VMX. The pfncache infrastructure maintains persistent mappings as long as the page GPA does not change, eliminating the memremap/memunmap overhead on every VM entry/exit cycle. Additionally, pfncache provides proper invalidation handling via mmu_notifier callbacks and memslots generation check, ensuring that mappings are correctly updated during both memslot updates and page migration events.
As an example, a microbenchmark using memslot_perf_test with 8192 memslots demonstrates huge improvements in nested VMX operations with unmanaged guest memory (this is a synthetic benchmark run on AWS EC2 Nitro instances, and the results are not representative of typical nested virtualization workloads):
Before After Improvement map: 26.12s 1.54s ~17x faster unmap: 40.00s 0.017s ~2353x faster unmap chunked: 10.07s 0.005s ~2014x faster
The series is organized as follows:
Patches 1-5 handle the L1 MSR bitmap page and system pages (APIC access, virtual APIC, and posted interrupt descriptor). Patch 1 converts the MSR bitmap to use gfn_to_pfn_cache. Patches 2-3 restore and complete "guest-uses-pfn" support in pfncache. Patch 4 converts the system pages to use gfn_to_pfn_cache. Patch 5 adds a selftest for cache invalidation and memslot updates.
Patches 6-7 add enlightened VMCS support. Patch 6 avoids accessing eVMCS fields after they are copied into the cached vmcs12 structure. Patch 7 converts eVMCS page mapping to use gfn_to_pfn_cache.
Patches 8-10 implement persistent nested context to handle L2 vCPU multiplexing and migration between L1 vCPUs. Patch 8 introduces the nested context management infrastructure. Patch 9 integrates pfncache with persistent nested context. Patch 10 adds a selftest for this L2 vCPU context switching.
v4:
- Rebase on kvm/next required additional vapic handling in patch 4 and a tiny fix in patch 5.
- Fix patch 9 to re-assign vcpu to pfncache if the nested context has been recycled, and to clear the vcpu context in free_nested().
v3:
- fixed warnings reported by kernel test robot in patches 7 and 8.
v2:
- Extended series to support enlightened VMCS (eVMCS).
- Added persistent nested context for improved L2 vCPU handling.
- Added additional selftests.
Fred Griffoul (10):
KVM: nVMX: Implement cache for L1 MSR bitmap
KVM: pfncache: Restore guest-uses-pfn support
KVM: x86: Add nested state validation for pfncache support
KVM: nVMX: Implement cache for L1 APIC pages
KVM: selftests: Add nested VMX APIC cache invalidation test
KVM: nVMX: Cache evmcs fields to ensure consistency during VM-entry
KVM: nVMX: Replace evmcs kvm_host_map with pfncache
KVM: x86: Add nested context management
KVM: nVMX: Use nested context for pfncache persistence
KVM: selftests: Add L2 vcpu context switch test
arch/x86/include/asm/kvm_host.h | 32 ++ arch/x86/include/uapi/asm/kvm.h | 2 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/nested.c | 199 +++++++ arch/x86/kvm/vmx/hyperv.c | 5 +- arch/x86/kvm/vmx/hyperv.h | 33 +- arch/x86/kvm/vmx/nested.c | 499 ++++++++++++++---- arch/x86/kvm/vmx/vmx.c | 8 + arch/x86/kvm/vmx/vmx.h | 16 +- arch/x86/kvm/x86.c | 19 +- include/linux/kvm_host.h | 34 +- include/linux/kvm_types.h | 1 + tools/testing/selftests/kvm/Makefile.kvm | 2 + .../selftests/kvm/x86/vmx_apic_update_test.c | 302 +++++++++++ .../selftests/kvm/x86/vmx_l2_switch_test.c | 416 +++++++++++++++ virt/kvm/kvm_main.c | 3 +- virt/kvm/kvm_mm.h | 6 +- virt/kvm/pfncache.c | 43 +- 18 files changed, 1496 insertions(+), 126 deletions(-) create mode 100644 arch/x86/kvm/nested.c create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_update_test.c create mode 100644 tools/testing/selftests/kvm/x86/vmx_l2_switch_test.c
base-commit: 0499add8efd72456514c6218c062911ccc922a99 --
2.43.0
PATCH v4 1/10 KVM: nVMX: Implement cache for L1 MSR bitmap
From: Fred Griffoul <[email protected]>
Introduce a gfn_to_pfn_cache to optimize L1 MSR bitmap access by replacing map/unmap operations. This optimization reduces overhead during L2 VM-entry where nested_vmx_prepare_msr_bitmap() merges L1's MSR intercepts with L0's requirements.
Current implementation using kvm_vcpu_map_readonly() and kvm_vcpu_unmap() creates significant performance impact, mostly with unmanaged guest memory.
The cache is initialized when entering VMX operation and deactivated when VMX operation ends.
--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/vmx.h | 2 ++ 2 files changed, 40 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/nested.c +38 -4
@@ -316,6 +316,34 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) vcpu->arch.regs_dirty =0 ; } +/* +* Maps a single guest page starting at @gpa and lock the cache for access. +*/ +static int nested_gpc_lock (struct gfn_to_pfn_cache *gpc,gpa_t gpa) +{ +int err; + +if (!PAGE_ALIGNED (gpa)) +return -EINVAL ; +retry: +read_lock (&gpc->lock); +if (!kvm_gpc_check (gpc,PAGE_SIZE ) || (gpc->gpa != gpa)) { +read_unlock (&gpc->lock); + err =kvm_gpc_activate (gpc, gpa,PAGE_SIZE ); +if (err) +return err; + + goto retry; + } + +return 0 ; +} + +static void nested_gpc_unlock (struct gfn_to_pfn_cache *gpc) +{ +read_unlock (&gpc->lock); +} +static void nested_put_vmcs12_pages (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu);
@@ -345,6 +373,9 @@ static void free_nested(struct kvm_vcpu *vcpu) vmx->nested.vmxon = false; vmx->nested.smm.vmxon = false; vmx->nested.vmxon_ptr =INVALID_GPA ; + +kvm_gpc_deactivate (&vmx->nested.msr_bitmap_cache); +free_vpid (vmx->nested.vpid02); vmx->nested.posted_intr_nv =-1 ; vmx->nested.current_vmptr =INVALID_GPA ;
@@ -629,7 +660,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,int msr;unsigned long *msr_bitmap_l1;unsigned long *msr_bitmap_l0 = vmx->nested.vmcs02.msr_bitmap; -struct kvm_host_map map; +struct gfn_to_pfn_cache *gpc;/* Nothing to do if the MSR bitmap is not in use. */ if (!cpu_has_vmx_msr_bitmap() ||
@@ -652,10 +683,11 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,return true; } -if (kvm_vcpu_map_readonly (vcpu,gpa_to_gfn (vmcs12->msr_bitmap), &map)) + gpc = &vmx->nested.msr_bitmap_cache; +if (nested_gpc_lock (gpc, vmcs12->msr_bitmap))return false; - msr_bitmap_l1 = (unsigned long *)map.hva; + msr_bitmap_l1 = (unsigned long *)gpc->khva;/* * To keep the control flow simple, pay eight 8-byte writes (sixteen
@@ -743,7 +775,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,nested_vmx_set_intercept_for_msr (vmx, msr_bitmap_l1, msr_bitmap_l0,MSR_IA32_PL3_SSP ,MSR_TYPE_RW ); -kvm_vcpu_unmap (vcpu, &map); +nested_gpc_unlock (gpc); vmx->nested.force_msr_bitmap_recalc = false;
@@ -5440,6 +5472,8 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) vmx->nested.vpid02 =allocate_vpid (); +kvm_gpc_init (&vmx->nested.msr_bitmap_cache, vcpu->kvm); + vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true;
arch/x86/kvm/vmx/vmx.h +2 -0
@@ -152,6 +152,8 @@ struct nested_vmx {struct loaded_vmcs vmcs02; +struct gfn_to_pfn_cache msr_bitmap_cache; +/* * Guest pages referred to in the vmcs02 with host-physical * pointers, so we must keep them pinned while L2 runs.
Signature · 1 line
2.43.0
Sean Christopherson
maintainerRe: [PATCH v4 01/10] KVM: nVMX: Implement cache for L1 MSR bitmap
Quoted 25 lines · Fred Griffoul
From: Fred Griffoul <[email protected]>
Introduce a
gfn_to_pfn_cacheto optimize L1 MSR bitmap access by replacing map/unmap operations. This optimization reduces overhead during L2 VM-entry wherenested_vmx_prepare_msr_bitmap()merges L1's MSR intercepts with L0's requirements.Current implementation using
kvm_vcpu_map_readonly()andkvm_vcpu_unmap()creates significant performance impact, mostly with unmanaged guest memory.The cache is initialized when entering VMX operation and deactivated when VMX operation ends.
Signed-off-by Fred Griffoul <[email protected]>--- arch/x86/kvm/vmx/nested.c | 42 +++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/vmx.h | 2 ++ 2 files changed, 40 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/nested.c +0
-0
arch/x86/kvm/vmx/nested.c static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) · line 322
Quoted 7 lines
arch/x86/kvm/vmx/nested.c +4
-0@@ -316,6 +316,34 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) vcpu->arch.regs_dirty =0 ; } +/* +* Maps a single guest page starting at @gpa and lock the cache for access. +*/ +static int nested_gpc_lock (struct gfn_to_pfn_cache *gpc,gpa_t gpa)
These helpers belong in virt/kvm/pfncache.c, not in vmx/nested.c. The names also need to be changed (which is super obvious when they land in pfncache.c). Nested locks are very much a thing in the kernel; even with the nVXM context, these read as "lock a GPC within another GPC critical section".
And this does waaaay more than just "lock" the gpc. E.g. by the end of the series this picks up:
err = kvm_gpc_activate(gpc, gpa, PAGE_SIZE);
if (err) {
/*
* Deactivate nested state caches to prevent
* kvm_gpc_invalid() from returning true in subsequent
* is_nested_state_invalid() calls. This prevents an
* infinite loop while entering guest mode.
*/
if (gpc->vcpu)
kvm_gpc_deactivate(gpc);
return err;
}That's a pretty gross hack, as is the addition and use of kvm_gpc_invalid() to detect that a vmcs12 page has been invalidated (though that may just be a naming issue).
The use of nested_gpc_lock() in nested_vmx_prepare_msr_bitmap() and nested_vmx_handle_enlightened_vmptrld() also feels forced. The usage there is _much_ more aligned with nested_gpc_lock_if_active(), in the sense that it's a temporary establishing a host mapping versus grabbing a pfn that gets shoved into a non-MMU structure.
arch/x86/kvm/vmx/nested.c static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) · line 330
Quoted 8 lines
arch/x86/kvm/vmx/nested.c +8
-0+{ +int err; + +if (!PAGE_ALIGNED (gpa)) +return -EINVAL ; +retry: +read_lock (&gpc->lock); +if (!kvm_gpc_check (gpc,PAGE_SIZE ) || (gpc->gpa != gpa)) {
This order of thsese checks is bizarre. It's not problematic per se, but it's jarring to read.
arch/x86/kvm/vmx/nested.c static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) · line 332
Quoted 2 lines
arch/x86/kvm/vmx/nested.c +2
-0+read_unlock (&gpc->lock); + err =kvm_gpc_activate (gpc, gpa,PAGE_SIZE );
And the use of kvm_gpc_activate() in the retry loop is also bizarre. It's "fine", but it makes the lifecycle of the code hard to follow.
The other thing that stands out in later patches is the error handling. I really dislike the number of unlock() calls that are being added.
The good news is, I think pretty much all of my complaints are symptoms of major gaps in the core gpc APIs. E.g. you're smushing largely unrelated things into nested_gpc_lock() because the retry loop is fugly. If we add better APIs, then the nVMX code shouldn't need to combine different concepts in weird ways, just to avoid copy+pasting an ugly loop.
Happily, the majority of that can be sorted out in advance of converting nVMX. E.g. instead of the stealtime conversion looking like so:
if (!read_trylock(&gpc->lock)) return; if (!kvm_gpc_check(gpc, sizeof(*st))) goto out_unlock_gpc; st = (struct kvm_steal_time *)gpc->khva; WRITE_ONCE(st->preempted, preempted); vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED; kvm_gpc_mark_dirty_in_slot(gpc);
out_unlock_gpc:
read_unlock(&gpc->lock);
it can instead be something like:
CLASS(gpc_try_map_local, st_map)(gpc, sizeof(st->preempted)); if (IS_ERR(st_map)) return; st = *st_map; WRITE_ONCE(st->preempted, KVM_VCPU_PREEMPTED); vcpu->arch.st.preempted = KVM_VCPU_PREEMPTED;
I have patches that appear to work. I'll post a new version of David's series[*] with the new APIs, and we can go from there (dropping the IRQ interactions makes adding the CLASS() stuff easier, and you'll have to rebase regardless).
arch/x86/kvm/vmx/nested.c static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs) · line 342
Quoted 10 lines
arch/x86/kvm/vmx/nested.c +10
-0+if (err) +return err; + + goto retry; + } + +return 0 ; +} + +static void nested_gpc_unlock (struct gfn_to_pfn_cache *gpc)
This is a completely superfluous helper. Looking ahead, many of the wrappers are similarly useless, and IMO do more harm than good. E.g. having bounce through multiple implementations for something like this:
if (!pi_test_and_clear_on(pi_desc)) {
nested_unlock_pi_desc(vmx);
return 0;
}is *really* frustrating when it's nothing more than:
if (!pi_test_and_clear_on(pi_desc)) {
read_unlock(&vmx->nested.pi_desc_cache->lock);
return 0;
}All of that should be a moot point with better APIs, so this really "in the future..." type of feedback.
PATCH v4 2/10 KVM: pfncache: Restore guest-uses-pfn support
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Restore functionality for guest page access tracking in pfncache, enabling automatic vCPU request generation when cache invalidation occurs through MMU notifier events.
This feature is critical for nested VMX operations where both KVM and L2 guest access guest-provided pages, such as APIC pages and posted interrupt descriptors.
This change:
- Reverts commit eefb85b3f031 ("KVM: Drop unused @may_block param from gfn_to_pfn_cache_invalidate_start()")
- Partially reverts commit a4bff3df5147 ("KVM: pfncache: remove KVM_GUEST_USES_PFN usage"). Adds kvm_gpc_init_for_vcpu() to initialize pfncache for guest mode access, instead of the usage-specific flag approach.
--- include/linux/kvm_host.h | 29 +++++++++++++++++++++++++- include/linux/kvm_types.h | 1 + virt/kvm/kvm_main.c | 3 ++- virt/kvm/kvm_mm.h | 6 ++++-- virt/kvm/pfncache.c | 43 ++++++++++++++++++++++++++++++++++++--- 5 files changed, 75 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +28 -1
@@ -1412,6 +1412,9 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *data, unsigned long len);void kvm_vcpu_mark_page_dirty (struct kvm_vcpu *vcpu,gfn_t gfn); +void __kvm_gpc_init (struct gfn_to_pfn_cache *gpc,struct kvm *kvm, +struct kvm_vcpu *vcpu); +/** * kvm_gpc_init - initialize gfn_to_pfn_cache. *
@@ -1422,7 +1425,11 @@ void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn);* immutable attributes. Note, the cache must be zero-allocated (or zeroed by * the caller before init). */ -void kvm_gpc_init (struct gfn_to_pfn_cache *gpc,struct kvm *kvm); + +static inline void kvm_gpc_init (struct gfn_to_pfn_cache *gpc,struct kvm *kvm) +{ +__kvm_gpc_init (gpc, kvm,NULL ); +}/** * kvm_gpc_activate - prepare a cached kernel mapping and HPA for a given guest
@@ -1504,6 +1511,26 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len);*/ void kvm_gpc_deactivate (struct gfn_to_pfn_cache *gpc); +/** +* kvm_gpc_init_for_vcpu - initialize gfn_to_pfn_cache for pin/unpin usage +* +* @gpc: struct gfn_to_pfn_cache object. +* @vcpu: vCPU that will pin and directly access this cache. +* @req: request to send when cache is invalidated while pinned. +* +* This sets up a gfn_to_pfn_cache for use by a vCPU that will directly access +* the cached physical address. When the cache is invalidated while pinned, +* the specified request will be sent to the associated vCPU to force cache +* refresh. +* +* Note, the cache must be zero-allocated (or zeroed by the caller before init). +*/ +static inline void kvm_gpc_init_for_vcpu (struct gfn_to_pfn_cache *gpc, +struct kvm_vcpu *vcpu) +{ +__kvm_gpc_init (gpc, vcpu->kvm, vcpu); +} +static inline bool kvm_gpc_is_gpa_active (struct gfn_to_pfn_cache *gpc) {return gpc->active && !kvm_is_error_gpa (gpc->gpa);
include/linux/kvm_types.h +1 -0
@@ -88,6 +88,7 @@ struct gfn_to_pfn_cache {struct kvm_memory_slot *memslot;struct kvm *kvm;struct list_head list; +struct kvm_vcpu *vcpu;rwlock_t lock;struct mutex refresh_lock;void *khva;
virt/kvm/kvm_main.c +2 -1
@@ -761,7 +761,8 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,* mn_active_invalidate_count (see above) instead of * mmu_invalidate_in_progress. */ -gfn_to_pfn_cache_invalidate_start (kvm, range->start, range->end); +gfn_to_pfn_cache_invalidate_start (kvm, range->start, range->end, + hva_range.may_block);/* * If one or more memslots were found and thus zapped, notify arch code
virt/kvm/kvm_mm.h +4 -2
@@ -58,11 +58,13 @@ kvm_pfn_t hva_to_pfn(struct kvm_follow_pfn *kfp);#ifdef CONFIG_HAVE_KVM_PFNCACHE void gfn_to_pfn_cache_invalidate_start (struct kvm *kvm,unsigned long start, -unsigned long end); +unsigned long end, +bool may_block);#else static inline void gfn_to_pfn_cache_invalidate_start (struct kvm *kvm,unsigned long start, -unsigned long end) +unsigned long end, +bool may_block) { }#endif /* HAVE_KVM_PFNCACHE */
virt/kvm/pfncache.c +40 -3
76 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v4 02/10] KVM: pfncache: Restore guest-uses-pfn support
Quoted 9 lines · Fred Griffoul
virt/kvm/pfncache.c +4
-0@@ -55,6 +70,27 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, read_unlock_irq(&gpc->lock); } spin_unlock(&kvm->gpc_lock); + +if (evict_vcpus) { +/* +* KVM needs to ensure the vCPU is fully out of guest context
virt/kvm/pfncache.c void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 86
Quoted 10 lines
virt/kvm/pfncache.c +10
-0+* before allowing the invalidation to continue. +*/ +unsigned int req =KVM_REQ_OUTSIDE_GUEST_MODE ; +bool called; + +/* +* If the OOM reaper is active, then all vCPUs should have +* been stopped already, so perform the request without +* KVM_REQUEST_WAIT and be sad if any needed to be IPI'd. +*/
I am unconvinced of the veracity of that claim. We get here via __oom_reap_task_mm() which has explicit comments about 'might start racing with the dying task'.
I think it's actually fine to use KVM_REQUEST_WAIT in theory... but then again, the task has already been ignoring SIGKILL for two seconds or the oom_reaper wouldn't even have been invoked AIUI, so maybe we should err on the side of caution by still doing this:
virt/kvm/pfncache.c void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 92
Quoted 6 lines
virt/kvm/pfncache.c +6
-0+if (!may_block) + req &= ~KVM_REQUEST_WAIT ; + + called =kvm_make_vcpus_request_mask (kvm, req, vcpu_bitmap); + +WARN_ON_ONCE (called && !may_block);
... but instead of warning, it should return -EINVAL in this case? The OOM reaper should call it to try again in 100ms. I guess we shouldn't clear the valid flag in that case? Because it does need to check the vCPUs again on the retry, not just skip on the basis that the invalidation already happened.
(Yes, this is a void function. It would need not to be, and the return would need to be propagated back through kvm_mmu_notifier_invalidate_range_start())
Sean Christopherson
maintainerRe: [PATCH v4 02/10] KVM: pfncache: Restore guest-uses-pfn support
Quoted 25 lines · David Woodhouse
Quoted 9 lines · Fred Griffoul
virt/kvm/pfncache.c +4
-0@@ -55,6 +70,27 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, read_unlock_irq(&gpc->lock); } spin_unlock(&kvm->gpc_lock); + +if (evict_vcpus) { +/* +* KVM needs to ensure the vCPU is fully out of guest context virt/kvm/pfncache.c void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long start, · line 86
Quoted 10 lines
virt/kvm/pfncache.c +10
-0+* before allowing the invalidation to continue. +*/ +unsigned int req =KVM_REQ_OUTSIDE_GUEST_MODE ; +bool called; + +/* +* If the OOM reaper is active, then all vCPUs should have +* been stopped already, so perform the request without +* KVM_REQUEST_WAIT and be sad if any needed to be IPI'd. +*/ I am unconvinced of the veracity of that claim. We get here via
__oom_reap_task_mm()which has explicit comments about 'might start racing with the dying task'.
Quoted 1 line
I think it's actually fine to use
KVM_REQUEST_WAITin theory...
Actually, it has to be "fine", because KVM already does KVM_REQUEST_WAIT for literally every releavnt OOM kill thanks to the kvm_flush_remote_tlbs() in kvm_mmu_notifier_invalidate_range_start() -> kvm_handle_hva_range(). So I would say don't worry about KVM_REQUEST_WAIT here; if we need to fix anything for the OOM case, then we need a more generic fix.
PATCH v4 3/10 KVM: x86: Add nested state validation for pfncache support
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Implement state validation for nested virtualization to enable pfncache support for L1 guest pages.
This adds a new nested_ops callback 'is_nested_state_invalid()' that detects when KVM needs to reload nested virtualization state. A KVM_REQ_GET_NESTED_STATE_PAGES request is triggered to reload affected pages before L2 execution when it detects invalid state. The callback monitors L1 guest pages during guest entry/exit while the vCPU runs in IN_GUEST_MODE.
Currently, VMX implementations return false, with full support planned for the next patch.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx/nested.c | 6 ++++++ arch/x86/kvm/x86.c | 14 +++++++++++++- 3 files changed, 20 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1956,6 +1956,7 @@ struct kvm_x86_nested_ops {struct kvm_nested_state __user *user_kvm_nested_state, struct kvm_nested_state *kvm_state);bool (*get_nested_state_pages)(struct kvm_vcpu *vcpu); +bool (*is_nested_state_invalid)(struct kvm_vcpu *vcpu);int (*write_log_dirty)(struct kvm_vcpu *vcpu,gpa_t l2_gpa); int (*enable_evmcs)(struct kvm_vcpu *vcpu,
arch/x86/kvm/vmx/nested.c +6 -0
@@ -3541,6 +3541,11 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)return true; } +static bool vmx_is_nested_state_invalid (struct kvm_vcpu *vcpu) +{ +return false; +} +static int nested_vmx_write_pml_buffer (struct kvm_vcpu *vcpu,gpa_t gpa) {struct vmcs12 *vmcs12;
@@ -7485,6 +7490,7 @@ struct kvm_x86_nested_ops vmx_nested_ops = { .get_state = vmx_get_nested_state, .set_state = vmx_set_nested_state, .get_nested_state_pages = vmx_get_nested_state_pages, + .is_nested_state_invalid = vmx_is_nested_state_invalid, .write_log_dirty = nested_vmx_write_pml_buffer, #ifdef CONFIG_KVM_HYPERV .enable_evmcs = nested_enable_evmcs,
arch/x86/kvm/x86.c +13 -1
@@ -2257,12 +2257,24 @@ int kvm_emulate_monitor(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_emulate_monitor); +static inline bool kvm_invalid_nested_state (struct kvm_vcpu *vcpu) +{ +if (is_guest_mode (vcpu) && + kvm_x86_ops.nested_ops->is_nested_state_invalid && + kvm_x86_ops.nested_ops->is_nested_state_invalid (vcpu)) { +kvm_make_request (KVM_REQ_GET_NESTED_STATE_PAGES , vcpu); +return true; + } +return false; +} +static inline bool kvm_vcpu_exit_request (struct kvm_vcpu *vcpu) {xfer_to_guest_mode_prepare ();return READ_ONCE (vcpu->mode) ==EXITING_GUEST_MODE || -kvm_request_pending (vcpu) ||xfer_to_guest_mode_work_pending (); +kvm_request_pending (vcpu) ||xfer_to_guest_mode_work_pending () || +kvm_invalid_nested_state (vcpu); }static fastpath_t __handle_fastpath_wrmsr (struct kvm_vcpu *vcpu,u32 msr,u64 data)
Signature · 1 line
2.43.0
PATCH v4 4/10 KVM: nVMX: Implement cache for L1 APIC pages
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Replace kvm_host_map usage with gfn_to_pfn_cache for L1 APIC virtualization pages (APIC access, virtual APIC, and posted interrupt descriptor pages) to improve performance with unmanaged guest memory.
The conversion involves several changes:
- Page loading in nested_get_vmcs12_pages(): load vmcs02 fields with pfncache PFNs after each cache has been checked and possibly activated or refreshed, during OUTSIDE_GUEST_MODE vCPU mode.
- Invalidation window handling: since nested_get_vmcs12_pages() runs in OUTSIDE_GUEST_MODE, there's a window where caches can be invalidated by MMU notifications before entering IN_GUEST_MODE. implement is_nested_state_invalid() callback to monitor cache validity between OUTSIDE_GUEST_MODE and IN_GUEST_MODE transitions. This triggers KVM_REQ_GET_NESTED_STATE_PAGES when needed.
- Cache access in event callbacks: the virtual APIC and posted interrupt descriptor pages are accessed by KVM in has_events() and check_events() nested_ops callbacks. These use the kernel HVA following the pfncache pattern of check/refresh, with both callbacks able to sleep if cache refresh is required.
This eliminates expensive memremap/memunmap cycles for each L2 VM entry/exit, providing substantial performance improvements when using unmanaged memory.
--- arch/x86/kvm/vmx/nested.c | 182 +++++++++++++++++++++++++++++--------- arch/x86/kvm/vmx/vmx.h | 8 +- include/linux/kvm_host.h | 5 ++ 3 files changed, 150 insertions(+), 45 deletions(-)
arch/x86/kvm/vmx/nested.c +141 -41
@@ -330,8 +330,18 @@ static int nested_gpc_lock(struct gfn_to_pfn_cache *gpc, gpa_t gpa)if (!kvm_gpc_check (gpc,PAGE_SIZE ) || (gpc->gpa != gpa)) {read_unlock (&gpc->lock); err =kvm_gpc_activate (gpc, gpa,PAGE_SIZE ); -if (err) +if (err) { +/* +* Deactivate nested state caches to prevent +* kvm_gpc_invalid() from returning true in subsequent +* is_nested_state_invalid() calls. This prevents an +* infinite loop while entering guest mode. +*/ +if (gpc->vcpu) +kvm_gpc_deactivate (gpc); +return err; + } goto retry; }
@@ -344,14 +354,64 @@ static void nested_gpc_unlock(struct gfn_to_pfn_cache *gpc)read_unlock (&gpc->lock); } -static void nested_put_vmcs12_pages (struct kvm_vcpu *vcpu) +static int nested_gpc_hpa (struct gfn_to_pfn_cache *gpc,gpa_t gpa,hpa_t *hpa) { -struct vcpu_vmx *vmx =to_vmx (vcpu); +int err; + + err =nested_gpc_lock (gpc, gpa); +if (err) +return err; + + *hpa =pfn_to_hpa (gpc->pfn); +nested_gpc_unlock (gpc); +return 0 ; +} + +static void *nested_gpc_lock_if_active (struct gfn_to_pfn_cache *gpc) +{ +if (!gpc) +return NULL ; +retry: +read_lock (&gpc->lock); +if (!gpc->active) { +read_unlock (&gpc->lock); +return NULL ; + } + +if (!kvm_gpc_check (gpc,PAGE_SIZE )) { +read_unlock (&gpc->lock); +if (kvm_gpc_refresh (gpc,PAGE_SIZE )) +return NULL ; + goto retry; + } + +return gpc->khva; +} + +static struct pi_desc *nested_lock_pi_desc (struct vcpu_vmx *vmx) +{ +u8 *pi_desc_page; + + pi_desc_page =nested_gpc_lock_if_active (vmx->nested.pi_desc_cache); +if (!pi_desc_page) +return NULL ; + +return (struct pi_desc *)(pi_desc_page + vmx->nested.pi_desc_offset); +} + +static void nested_unlock_pi_desc (struct vcpu_vmx *vmx) +{ +nested_gpc_unlock (vmx->nested.pi_desc_cache); +} + +static void *nested_lock_vapic (struct vcpu_vmx *vmx) +{ +return nested_gpc_lock_if_active (vmx->nested.virtual_apic_cache); +} -kvm_vcpu_unmap (vcpu, &vmx->nested.apic_access_page_map); -kvm_vcpu_unmap (vcpu, &vmx->nested.virtual_apic_map); -kvm_vcpu_unmap (vcpu, &vmx->nested.pi_desc_map); - vmx->nested.pi_desc =NULL ; +static void nested_unlock_vapic (struct vcpu_vmx *vmx) +{ +nested_gpc_unlock (vmx->nested.virtual_apic_cache); }/*
@@ -374,6 +434,9 @@ static void free_nested(struct kvm_vcpu *vcpu) vmx->nested.smm.vmxon = false; vmx->nested.vmxon_ptr =INVALID_GPA ; +kvm_gpc_deactivate (&vmx->nested.pi_desc_cache); +kvm_gpc_deactivate (&vmx->nested.virtual_apic_cache); +kvm_gpc_deactivate (&vmx->nested.apic_access_page_cache);kvm_gpc_deactivate (&vmx->nested.msr_bitmap_cache);free_vpid (vmx->nested.vpid02);
@@ -390,8 +453,6 @@ static void free_nested(struct kvm_vcpu *vcpu)kfree (vmx->nested.cached_shadow_vmcs12); vmx->nested.cached_shadow_vmcs12 =NULL ; -nested_put_vmcs12_pages (vcpu); -kvm_mmu_free_roots (vcpu->kvm, &vcpu->arch.guest_mmu,KVM_MMU_ROOTS_ALL );nested_release_evmcs (vcpu);
@@ -3112,8 +3173,9 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,static int nested_vmx_check_controls_late (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12) { -void *vapic =to_vmx (vcpu)->nested.virtual_apic_map.hva; -u32 vtpr = vapic ? (*(u32 *)(vapic +APIC_TASKPRI )) >>4 :0 ; +struct vcpu_vmx *vmx =to_vmx (vcpu); +void *vapic; +u32 vtpr =0 ;/* * Don't bother with the consistency checks if KVM isn't configured to
@@ -3130,6 +3192,12 @@ static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,if (!warn_on_missed_cc)return 0 ; + vapic =nested_lock_vapic (vmx); +if (vapic) { + vtpr = (*(u32 *)(vapic +APIC_TASKPRI )) >>4 ; +nested_unlock_vapic (vmx); + } + if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) && nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) && !nested_cpu_has_vid(vmcs12) &&
@@ -3430,7 +3498,8 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu); -struct kvm_host_map *map; +struct gfn_to_pfn_cache *gpc; +hpa_t hpa;if (!vcpu->arch.pdptrs_from_userspace && !nested_cpu_has_ept (vmcs12) &&is_pae_paging (vcpu)) {
@@ -3445,10 +3514,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)if (nested_cpu_has2 (vmcs12,SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES )) { - map = &vmx->nested.apic_access_page_map; + gpc = &vmx->nested.apic_access_page_cache; -if (!kvm_vcpu_map (vcpu,gpa_to_gfn (vmcs12->apic_access_addr), map)) { -vmcs_write64 (APIC_ACCESS_ADDR ,pfn_to_hpa (map->pfn)); +if (!nested_gpc_hpa (gpc, vmcs12->apic_access_addr, &hpa)) { +vmcs_write64 (APIC_ACCESS_ADDR , hpa); }else {pr_debug_ratelimited ("%s: no backing for APIC-access address in vmcs12\n" , __func__);
@@ -3461,10 +3530,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) }if (nested_cpu_has (vmcs12,CPU_BASED_TPR_SHADOW )) { - map = &vmx->nested.virtual_apic_map; + gpc = &vmx->nested.virtual_apic_cache; -if (!kvm_vcpu_map (vcpu,gpa_to_gfn (vmcs12->virtual_apic_page_addr), map)) { -vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR ,pfn_to_hpa (map->pfn)); +if (!nested_gpc_hpa (gpc, vmcs12->virtual_apic_page_addr, &hpa)) { +vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR , hpa); }else if (nested_cpu_has (vmcs12,CPU_BASED_CR8_LOAD_EXITING ) &&nested_cpu_has (vmcs12,CPU_BASED_CR8_STORE_EXITING ) && !nested_cpu_has2 (vmcs12,SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES )) {
@@ -3487,14 +3556,12 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) }if (nested_cpu_has_posted_intr (vmcs12)) { - map = &vmx->nested.pi_desc_map; + gpc = &vmx->nested.pi_desc_cache; -if (!kvm_vcpu_map (vcpu,gpa_to_gfn (vmcs12->posted_intr_desc_addr), map)) { - vmx->nested.pi_desc = - (struct pi_desc *)(((void *)map->hva) + -offset_in_page (vmcs12->posted_intr_desc_addr)); +if (!nested_gpc_hpa (gpc, vmcs12->posted_intr_desc_addr &PAGE_MASK , &hpa)) { + vmx->nested.pi_desc_offset =offset_in_page (vmcs12->posted_intr_desc_addr);vmcs_write64 (POSTED_INTR_DESC_ADDR , -pfn_to_hpa (map->pfn) +offset_in_page (vmcs12->posted_intr_desc_addr)); + hpa +offset_in_page (vmcs12->posted_intr_desc_addr)); }else {/* * Defer the KVM_INTERNAL_EXIT until KVM tries to
@@ -3502,7 +3569,6 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)* descriptor. (Note that KVM may do this when it * should not, per the architectural specification.) */ - vmx->nested.pi_desc =NULL ;pin_controls_clearbit (vmx,PIN_BASED_POSTED_INTR ); } }
@@ -3543,7 +3609,16 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)static bool vmx_is_nested_state_invalid (struct kvm_vcpu *vcpu) { -return false; +struct vcpu_vmx *vmx =to_vmx (vcpu); + +/* +* @vcpu is in IN_GUEST_MODE, eliminating the need for individual gpc +* locks. Since kvm_gpc_invalid() doesn't verify gpc memslot +* generation, we can also skip acquiring the srcu lock. +*/ +return kvm_gpc_invalid (&vmx->nested.apic_access_page_cache) || +kvm_gpc_invalid (&vmx->nested.virtual_apic_cache) || +kvm_gpc_invalid (&vmx->nested.pi_desc_cache); }static int nested_vmx_write_pml_buffer (struct kvm_vcpu *vcpu,gpa_t gpa)
@@ -4043,6 +4118,7 @@ void nested_mark_vmcs12_pages_dirty(struct kvm_vcpu *vcpu)static int vmx_complete_nested_posted_interrupt (struct kvm_vcpu *vcpu) {struct vcpu_vmx *vmx =to_vmx (vcpu); +struct pi_desc *pi_desc;int max_irr;void *vapic_page;u16 status;
@@ -4050,22 +4126,29 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)if (!vmx->nested.pi_pending)return 0 ; -if (!vmx->nested.pi_desc) + pi_desc =nested_lock_pi_desc (vmx); +if (!pi_desc) goto mmio_needed; vmx->nested.pi_pending = false; -if (!pi_test_and_clear_on (vmx->nested.pi_desc)) +if (!pi_test_and_clear_on (pi_desc)) { +nested_unlock_pi_desc (vmx);return 0 ; + } - max_irr =pi_find_highest_vector (vmx->nested.pi_desc); + max_irr =pi_find_highest_vector (pi_desc);if (max_irr >0 ) { - vapic_page = vmx->nested.virtual_apic_map.hva; -if (!vapic_page) + vapic_page =nested_lock_vapic (vmx); +if (!vapic_page) { +nested_unlock_pi_desc (vmx); goto mmio_needed; + } + +__kvm_apic_update_irr (pi_desc->pir, vapic_page, &max_irr); + +nested_unlock_vapic (vmx); -__kvm_apic_update_irr (vmx->nested.pi_desc->pir, - vapic_page, &max_irr); status =vmcs_read16 (GUEST_INTR_STATUS );if ((u8 )max_irr > ((u8 )status &0xff )) { status &= ~0xff ;
@@ -4074,6 +4157,7 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu) } } +nested_unlock_pi_desc (vmx);nested_mark_vmcs12_pages_dirty (vcpu);return 0 ;
@@ -4193,8 +4277,10 @@ static bool nested_vmx_preemption_timer_pending(struct kvm_vcpu *vcpu)static bool vmx_has_nested_events (struct kvm_vcpu *vcpu,bool for_injection) {struct vcpu_vmx *vmx =to_vmx (vcpu); -void *vapic = vmx->nested.virtual_apic_map.hva; +struct pi_desc *pi_desc;int max_irr, vppr; +void *vapic; +bool res = false;if (nested_vmx_preemption_timer_pending (vcpu) || vmx->nested.mtf_pending)
@@ -4213,23 +4299,33 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection) __vmx_interrupt_blocked(vcpu))return false; + vapic =nested_lock_vapic (vmx);if (!vapic)return false; vppr = *((u32 *)(vapic +APIC_PROCPRI )); +nested_unlock_vapic (vmx); + max_irr =vmx_get_rvi ();if ((max_irr &0xf0 ) > (vppr &0xf0 ))return true; -if (vmx->nested.pi_pending && vmx->nested.pi_desc && -pi_test_on (vmx->nested.pi_desc)) { - max_irr =pi_find_highest_vector (vmx->nested.pi_desc); -if (max_irr >0 && (max_irr &0xf0 ) > (vppr &0xf0 )) -return true; +if (vmx->nested.pi_pending) { + pi_desc =nested_lock_pi_desc (vmx); +if (!pi_desc) +return false; + +if (pi_test_on (pi_desc)) { + max_irr =pi_find_highest_vector (pi_desc); +if (max_irr >0 && (max_irr &0xf0 ) > (vppr &0xf0 )) + res = true; + } + +nested_unlock_pi_desc (vmx); } -return false; +return res; }/*
@@ -5194,7 +5290,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_update_cpu_dirty_logging (vcpu); } -nested_put_vmcs12_pages (vcpu); +nested_mark_vmcs12_pages_dirty (vcpu);if (vmx->nested.reload_vmcs01_apic_access_page) { vmx->nested.reload_vmcs01_apic_access_page = false;
@@ -5479,6 +5575,10 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu)kvm_gpc_init (&vmx->nested.msr_bitmap_cache, vcpu->kvm); +kvm_gpc_init_for_vcpu (&vmx->nested.apic_access_page_cache, vcpu); +kvm_gpc_init_for_vcpu (&vmx->nested.virtual_apic_cache, vcpu); +kvm_gpc_init_for_vcpu (&vmx->nested.pi_desc_cache, vcpu); + vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true;
arch/x86/kvm/vmx/vmx.h +4 -4
@@ -158,11 +158,11 @@ struct nested_vmx {* Guest pages referred to in the vmcs02 with host-physical * pointers, so we must keep them pinned while L2 runs. */ -struct kvm_host_map apic_access_page_map; -struct kvm_host_map virtual_apic_map; -struct kvm_host_map pi_desc_map; +struct gfn_to_pfn_cache apic_access_page_cache; +struct gfn_to_pfn_cache virtual_apic_cache; +struct gfn_to_pfn_cache pi_desc_cache; -struct pi_desc *pi_desc; +u64 pi_desc_offset;bool pi_pending;u16 posted_intr_nv;
include/linux/kvm_host.h +5 -0
@@ -1541,6 +1541,11 @@ static inline bool kvm_gpc_is_hva_active(struct gfn_to_pfn_cache *gpc)return gpc->active &&kvm_is_error_gpa (gpc->gpa); } +static inline bool kvm_gpc_invalid (struct gfn_to_pfn_cache *gpc) +{ +return gpc->active && !gpc->valid; +} +void kvm_sigset_activate (struct kvm_vcpu *vcpu);void kvm_sigset_deactivate (struct kvm_vcpu *vcpu);
Signature · 1 line
2.43.0
Sean Christopherson
maintainerRe: [PATCH v4 04/10] KVM: nVMX: Implement cache for L1 APIC pages
Quoted 29 lines · Fred Griffoul
From: Fred Griffoul <[email protected]>
Replace
kvm_host_mapusage withgfn_to_pfn_cachefor L1 APIC virtualization pages (APIC access, virtual APIC, and posted interrupt descriptor pages) to improve performance with unmanaged guest memory.The conversion involves several changes:
- Page loading in
nested_get_vmcs12_pages(): load vmcs02 fields with pfncache PFNs after each cache has been checked and possibly activated or refreshed, duringOUTSIDE_GUEST_MODEvCPU mode.- Invalidation window handling: since
nested_get_vmcs12_pages()runs inOUTSIDE_GUEST_MODE, there's a window where caches can be invalidated by MMU notifications before enteringIN_GUEST_MODE. implementis_nested_state_invalid()callback to monitor cache validity betweenOUTSIDE_GUEST_MODEandIN_GUEST_MODEtransitions. This triggersKVM_REQ_GET_NESTED_STATE_PAGESwhen needed.- Cache access in event callbacks: the virtual APIC and posted interrupt descriptor pages are accessed by KVM in
has_events()andcheck_events()nested_opscallbacks. These use the kernel HVA following the pfncache pattern of check/refresh, with both callbacks able to sleep if cache refresh is required.This eliminates expensive memremap/memunmap cycles for each L2 VM entry/exit, providing substantial performance improvements when using unmanaged memory.
Quoted 6 lines
Signed-off-by Fred Griffoul <[email protected]>--- arch/x86/kvm/vmx/nested.c | 182 +++++++++++++++++++++++++++++--------- arch/x86/kvm/vmx/vmx.h | 8 +- include/linux/kvm_host.h | 5 ++ 3 files changed, 150 insertions(+), 45 deletions(-)
Please split this up; one "cache" per patch. It's a bit gratutious, but the usage of the vapic and pid structures is just complex enough to make this more than a straightforward, "boring" conversion.
Quoted 13 lines
arch/x86/kvm/vmx/nested.c +3
-2@@ -3112,8 +3173,9 @@ static int nested_vmx_check_controls(struct kvm_vcpu *vcpu,static int nested_vmx_check_controls_late (struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12) { -void *vapic =to_vmx (vcpu)->nested.virtual_apic_map.hva; -u32 vtpr = vapic ? (*(u32 *)(vapic +APIC_TASKPRI )) >>4 :0 ; +struct vcpu_vmx *vmx =to_vmx (vcpu); +void *vapic; +u32 vtpr =0 ;/* * Don't bother with the consistency checks if KVM isn't configured to
arch/x86/kvm/vmx/nested.c static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu, · line 3198
Quoted 7 lines
arch/x86/kvm/vmx/nested.c +4
-0@@ -3130,6 +3192,12 @@ static int nested_vmx_check_controls_late(struct kvm_vcpu *vcpu,if (!warn_on_missed_cc)return 0 ; + vapic =nested_lock_vapic (vmx); +if (vapic) { + vtpr = (*(u32 *)(vapic +APIC_TASKPRI )) >>4 ; +nested_unlock_vapic (vmx);
This is *very* dangerous. vapic will still be a legal kernel pointer, and the code _relies_ on it to be non-NULL, but the pointer could become stale at any time. Happily, my CLASS() approach makes this a non-issue.
Quoted 5 lines
arch/x86/kvm/vmx/nested.c +2
-0+ } + if ((exec_controls_get(to_vmx(vcpu)) & CPU_BASED_TPR_SHADOW) && nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW) && !nested_cpu_has_vid(vmcs12) &&
...
arch/x86/kvm/vmx/nested.c static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) · line 3615
Quoted 9 lines
arch/x86/kvm/vmx/nested.c +4
-1@@ -3543,7 +3609,16 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)static bool vmx_is_nested_state_invalid (struct kvm_vcpu *vcpu) { -return false; +struct vcpu_vmx *vmx =to_vmx (vcpu); + +/* +* @vcpu is in IN_GUEST_MODE, eliminating the need for individual gpc
This is straight up wrong.
gpc->active is effectively protected by vcpu->mutex.
And for gpc->valid, it's not completely wrong, but it's partially wrong and definitely misleading. It too is protected by vcpu->mutex, but only for %false => %true transitions. And even for %true => %false transitions, IN_GUEST_MODE doesn't exactly provide safety, rather it ensures that KVM will to kick the vCPU out of the guest before completing the invalidation.
But that's all moot, because the very existence of this code feels wrong. KVM should be sending a "no action" request and then requiring vendor code to check out-of-band data to proxy that into a "real" request.
Presumably older code that you're reverting used KVM_REQ_OUTSIDE_GUEST_MODE; that doesn't necessary mean it was a good idea, it got yanked out for a reason :-)
IMO, the least awful way to deal with this is to tag KVM_REQ_GET_NESTED_STATE_PAGES with KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP, and then pass the specific request that needs to be made when invalidating a vCPU-mapped gpc as part of __kvm_gpc_init() (though maybe call it vcpu_gpc_init() instead?). That way KVM doesn't need this wonky request => kvm_gpc_invalid() => request proxying, and we should be avoid to entirely avoid the confusingly-named kvm_gpc_invalid().
PATCH v4 5/10 KVM: selftests: Add nested VMX APIC cache invalidation test
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Introduce selftest to verify nested VMX APIC virtualization page cache invalidation and refresh mechanisms for pfncache implementation.
The test exercises the nested VMX APIC cache invalidation path through:
- L2 guest setup: creates a nested environment where L2 accesses the APIC access page that is cached by KVM using pfncache.
- Cache invalidation triggers: a separate update thread periodically invalidates the cached pages using either:
- madvise(MADV_DONTNEED) to trigger MMU notifications.
- vm_mem_region_move() to trigger memslot changes.
The test validates that:
- L2 can successfully access APIC page before and after invalidation.
- KVM properly handles cache refresh without guest-visible errors.
- Both MMU notification and memslot change invalidation paths work correctly.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/vmx_apic_update_test.c | 302 ++++++++++++++++++ 2 files changed, 303 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -139,6 +139,7 @@ TEST_GEN_PROGS_x86 += x86/max_vcpuid_cap_test TEST_GEN_PROGS_x86 += x86/triple_fault_event_test TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test TEST_GEN_PROGS_x86 += x86/aperfmperf_test +TEST_GEN_PROGS_x86 += x86/vmx_apic_update_test TEST_GEN_PROGS_x86 += access_tracking_perf_test TEST_GEN_PROGS_x86 += coalesced_io_test TEST_GEN_PROGS_x86 += dirty_log_perf_test
tools/testing/selftests/kvm/x86/vmx_apic_update_test.c new file +302 -0
302 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v4 6/10 KVM: nVMX: Cache evmcs fields to ensure consistency during VM-entry
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Cache enlightened VMCS control fields to prevent TOCTOU races where the guest could modify hv_clean_fields or hv_enlightenments_control between multiple accesses during nested VM-entry.
The cached values ensure consistent behavior across:
- The evmcs-to-vmcs12 copy operations
- MSR bitmap validation
- Clean field checks in prepare_vmcs02_rare()
This eliminates potential guest-induced inconsistencies in nested virtualization state management.
--- arch/x86/kvm/vmx/hyperv.c | 5 ++-- arch/x86/kvm/vmx/hyperv.h | 20 +++++++++++++ arch/x86/kvm/vmx/nested.c | 62 ++++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 5 +++- 4 files changed, 65 insertions(+), 27 deletions(-)
arch/x86/kvm/vmx/hyperv.c +2 -3
@@ -213,12 +213,11 @@ bool nested_evmcs_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu); -struct hv_enlightened_vmcs *evmcs = vmx->nested.hv_evmcs; -if (!hv_vcpu || !evmcs) +if (!hv_vcpu || !nested_vmx_is_evmptr12_valid (vmx))return false; -if (!evmcs->hv_enlightenments_control.nested_flush_hypercall) +if (!vmx->nested.hv_flush_hypercall)return false;return hv_vcpu->vp_assist_page.nested_control.features.directhypercall;
arch/x86/kvm/vmx/hyperv.h +20 -0
@@ -52,6 +52,16 @@ static inline bool guest_cpu_cap_has_evmcs(struct kvm_vcpu *vcpu)to_vmx (vcpu)->nested.enlightened_vmcs_enabled; } +static inline u32 nested_evmcs_clean_fields (struct vcpu_vmx *vmx) +{ +return vmx->nested.hv_clean_fields; +} + +static inline bool nested_evmcs_msr_bitmap (struct vcpu_vmx *vmx) +{ +return vmx->nested.hv_msr_bitmap; +} +u64 nested_get_evmptr (struct kvm_vcpu *vcpu);uint16_t nested_get_evmcs_version (struct kvm_vcpu *vcpu); int nested_enable_evmcs(struct kvm_vcpu *vcpu,
@@ -85,6 +95,16 @@ static inline struct hv_enlightened_vmcs *nested_vmx_evmcs(struct vcpu_vmx *vmx) {return NULL ; } + +static inline u32 nested_evmcs_clean_fields (struct vcpu_vmx *vmx) +{ +return 0 ; +} + +static inline bool nested_evmcs_msr_bitmap (struct vcpu_vmx *vmx) +{ +return false; +}#endif #endif /* __KVM_X86_VMX_HYPERV_H */
arch/x86/kvm/vmx/nested.c +39 -23
139 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +4 -1
@@ -205,8 +205,11 @@ struct nested_vmx {#ifdef CONFIG_KVM_HYPERV gpa_t hv_evmcs_vmptr; -struct kvm_host_map hv_evmcs_map; +u32 hv_clean_fields; +bool hv_msr_bitmap; +bool hv_flush_hypercall;struct hv_enlightened_vmcs *hv_evmcs; +struct kvm_host_map hv_evmcs_map;#endif };
Signature · 1 line
2.43.0
Vitaly Kuznetsov
maintainerRe: [PATCH v4 06/10] KVM: nVMX: Cache evmcs fields to ensure consistency during VM-entry
Quoted 6 lines · Fred Griffoul
From: Fred Griffoul <[email protected]>
Cache enlightened VMCS control fields to prevent TOCTOU races where the guest could modify
hv_clean_fieldsorhv_enlightenments_controlbetween multiple accesses during nested VM-entry.
Quoted 7 lines
The cached values ensure consistent behavior across:
- The evmcs-to-vmcs12 copy operations
- MSR bitmap validation
- Clean field checks inprepare_vmcs02_rare()This eliminates potential guest-induced inconsistencies in nested virtualization state management.
Could you please split the commit to simplify the review and bughunting if (when?) issues are discovered? In particular, I would've split this into at least:
- hv_flush_hypercall caching/handling
- hv_msr_bitmap caching/handling
- nested_vmx_is_evmptr12_valid() instead of vmx->nested.hv_evmcs checks
- adding 'bool copy' argument to nested_vmx_handle_enlightened_vmptrld() and moving copy_enlightened_to_vmcs12() there from nested_vmx_run()
- hv_clean_fields caching/handling.
I want to be extra careful with clean fields. I may had forgotten the details since I've implemented it but I still remember some pain over inconsistent state upon migrations, namely
commit d6bf71a18c74de61548ddad44ff95306fe85f829
Author: Vitaly Kuznetsov <[email protected]> Date: Wed May 26 15:20:22 2021 +0200
KVM: nVMX: Ignore 'hv_clean_fields' data when eVMCS data is copied in vmx_get_nested_state()
Unfortunatelly, I don't think we have good test coverage for all this in 'hyperv_evmcs' selftest :-( I've seen you added evmcs support to 'vmx_l2_switch_test' in this series, thanks a lot for doing that!
Quoted 237 lines
Signed-off-by Fred Griffoul <[email protected]>--- arch/x86/kvm/vmx/hyperv.c | 5 ++-- arch/x86/kvm/vmx/hyperv.h | 20 +++++++++++++ arch/x86/kvm/vmx/nested.c | 62 ++++++++++++++++++++++++--------------- arch/x86/kvm/vmx/vmx.h | 5 +++- 4 files changed, 65 insertions(+), 27 deletions(-)arch/x86/kvm/vmx/hyperv.c +2
-3@@ -213,12 +213,11 @@ bool nested_evmcs_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu); -struct hv_enlightened_vmcs *evmcs = vmx->nested.hv_evmcs; -if (!hv_vcpu || !evmcs) +if (!hv_vcpu || !nested_vmx_is_evmptr12_valid (vmx))return false; -if (!evmcs->hv_enlightenments_control.nested_flush_hypercall) +if (!vmx->nested.hv_flush_hypercall)return false;return hv_vcpu->vp_assist_page.nested_control.features.directhypercall;arch/x86/kvm/vmx/hyperv.h +20
-0@@ -52,6 +52,16 @@ static inline bool guest_cpu_cap_has_evmcs(struct kvm_vcpu *vcpu)to_vmx (vcpu)->nested.enlightened_vmcs_enabled; } +static inline u32 nested_evmcs_clean_fields (struct vcpu_vmx *vmx) +{ +return vmx->nested.hv_clean_fields; +} + +static inline bool nested_evmcs_msr_bitmap (struct vcpu_vmx *vmx) +{ +return vmx->nested.hv_msr_bitmap; +} +u64 nested_get_evmptr (struct kvm_vcpu *vcpu);uint16_t nested_get_evmcs_version (struct kvm_vcpu *vcpu); int nested_enable_evmcs(struct kvm_vcpu *vcpu,@@ -85,6 +95,16 @@ static inline struct hv_enlightened_vmcs *nested_vmx_evmcs(struct vcpu_vmx *vmx) {return NULL ; } + +static inline u32 nested_evmcs_clean_fields (struct vcpu_vmx *vmx) +{ +return 0 ; +} + +static inline bool nested_evmcs_msr_bitmap (struct vcpu_vmx *vmx) +{ +return false; +}#endif #endif /* __KVM_X86_VMX_HYPERV_H */ arch/x86/kvm/vmx/nested.c +39
-23@@ -236,6 +236,9 @@ static inline void nested_release_evmcs(struct kvm_vcpu *vcpu)kvm_vcpu_unmap (vcpu, &vmx->nested.hv_evmcs_map); vmx->nested.hv_evmcs =NULL ; vmx->nested.hv_evmcs_vmptr =EVMPTR_INVALID ; + vmx->nested.hv_clean_fields =0 ; + vmx->nested.hv_msr_bitmap = false; + vmx->nested.hv_flush_hypercall = false;if (hv_vcpu) { hv_vcpu->nested.pa_page_gpa =INVALID_GPA ;@@ -737,10 +740,10 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,* and tells KVM (L0) there were no changes in MSR bitmap for L2. */ if (!vmx->nested.force_msr_bitmap_recalc) { -struct hv_enlightened_vmcs *evmcs =nested_vmx_evmcs (vmx); - -if (evmcs && evmcs->hv_enlightenments_control.msr_bitmap && - evmcs->hv_clean_fields &HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP ) +if (nested_vmx_is_evmptr12_valid (vmx) && +nested_evmcs_msr_bitmap (vmx) && + (nested_evmcs_clean_fields (vmx) + &HV_VMX_ENLIGHTENED_CLEAN_FIELD_MSR_BITMAP ))return true; }@@ -2214,10 +2217,11 @@ static void copy_vmcs12_to_enlightened(struct vcpu_vmx *vmx)* instruction. */ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld ( -struct kvm_vcpu *vcpu,bool from_launch) +struct kvm_vcpu *vcpu,bool from_launch,bool copy) {#ifdef CONFIG_KVM_HYPERV struct vcpu_vmx *vmx =to_vmx (vcpu); +struct hv_enlightened_vmcs *evmcs;bool evmcs_gpa_changed = false;u64 evmcs_gpa;@@ -2297,6 +2301,22 @@ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld( vmx->nested.force_msr_bitmap_recalc = true; } +/* Cache evmcs fields to avoid reading evmcs after copy to vmcs12 */ + evmcs = vmx->nested.hv_evmcs; + vmx->nested.hv_clean_fields = evmcs->hv_clean_fields; + vmx->nested.hv_flush_hypercall = evmcs->hv_enlightenments_control.nested_flush_hypercall; + vmx->nested.hv_msr_bitmap = evmcs->hv_enlightenments_control.msr_bitmap; + +if (copy) { +struct vmcs12 *vmcs12 =get_vmcs12 (vcpu); + +if (likely (!vmcs12->hdr.shadow_vmcs)) { +copy_enlightened_to_vmcs12 (vmx, vmx->nested.hv_clean_fields); +/* Enlightened VMCS doesn't have launch state */ + vmcs12->launch_state = !from_launch; + } + } +return EVMPTRLD_SUCCEEDED ;#else return EVMPTRLD_DISABLED ;@@ -2655,10 +2675,12 @@ static void vmcs_write_cet_state(struct kvm_vcpu *vcpu, u64 s_cet,static void prepare_vmcs02_rare (struct vcpu_vmx *vmx,struct vmcs12 *vmcs12) { -struct hv_enlightened_vmcs *hv_evmcs =nested_vmx_evmcs (vmx); +u32 hv_clean_fields =0 ; -if (!hv_evmcs || !(hv_evmcs->hv_clean_fields & -HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP2 )) { +if (nested_vmx_is_evmptr12_valid (vmx)) + hv_clean_fields =nested_evmcs_clean_fields (vmx); + +if (!(hv_clean_fields &HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP2 )) {vmcs_write16 (GUEST_ES_SELECTOR , vmcs12->guest_es_selector);vmcs_write16 (GUEST_CS_SELECTOR , vmcs12->guest_cs_selector);@@ -2700,8 +2722,7 @@ static void prepare_vmcs02_rare(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)vmx_segment_cache_clear (vmx); } -if (!hv_evmcs || !(hv_evmcs->hv_clean_fields & -HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 )) { +if (!(hv_clean_fields &HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 )) {vmcs_write32 (GUEST_SYSENTER_CS , vmcs12->guest_sysenter_cs);vmcs_writel (GUEST_PENDING_DBG_EXCEPTIONS , vmcs12->guest_pending_dbg_exceptions);@@ -2792,7 +2813,7 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, enum vm_entry_failure_code *entry_failure_code) {struct vcpu_vmx *vmx =to_vmx (vcpu); -struct hv_enlightened_vmcs *evmcs =nested_vmx_evmcs (vmx); +struct hv_enlightened_vmcs *evmcs;bool load_guest_pdptrs_vmcs12 = false;if (vmx->nested.dirty_vmcs12 ||nested_vmx_is_evmptr12_valid (vmx)) {@@ -2800,7 +2821,8 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, vmx->nested.dirty_vmcs12 = false; load_guest_pdptrs_vmcs12 = !nested_vmx_is_evmptr12_valid (vmx) || - !(evmcs->hv_clean_fields &HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 ); + !(nested_evmcs_clean_fields (vmx) + &HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1 ); } if (vmx->nested.nested_run_pending &&@@ -2929,7 +2951,8 @@ static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12,* bits when it changes a field in eVMCS. Mark all fields as clean * here. */ -if (nested_vmx_is_evmptr12_valid (vmx)) + evmcs =nested_vmx_evmcs (vmx); +if (evmcs) evmcs->hv_clean_fields |=HV_VMX_ENLIGHTENED_CLEAN_FIELD_ALL ;return 0 ;@@ -3477,7 +3500,7 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)if (guest_cpu_cap_has_evmcs (vcpu) && vmx->nested.hv_evmcs_vmptr ==EVMPTR_MAP_PENDING ) {enum nested_evmptrld_status evmptrld_status = -nested_vmx_handle_enlightened_vmptrld (vcpu, false); +nested_vmx_handle_enlightened_vmptrld (vcpu, false, false);if (evmptrld_status ==EVMPTRLD_VMFAIL || evmptrld_status ==EVMPTRLD_ERROR )@@ -3867,7 +3890,7 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)if (!nested_vmx_check_permission (vcpu))return 1 ; - evmptrld_status =nested_vmx_handle_enlightened_vmptrld (vcpu, launch); + evmptrld_status =nested_vmx_handle_enlightened_vmptrld (vcpu, launch, true);if (evmptrld_status ==EVMPTRLD_ERROR ) {kvm_queue_exception (vcpu,UD_VECTOR );return 1 ;@@ -3893,15 +3916,8 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)if (CC (vmcs12->hdr.shadow_vmcs))return nested_vmx_failInvalid (vcpu); -if (nested_vmx_is_evmptr12_valid (vmx)) { -struct hv_enlightened_vmcs *evmcs =nested_vmx_evmcs (vmx); - -copy_enlightened_to_vmcs12 (vmx, evmcs->hv_clean_fields); -/* Enlightened VMCS doesn't have launch state */ - vmcs12->launch_state = !launch; - }else if (enable_shadow_vmcs) { +if (!nested_vmx_is_evmptr12_valid (vmx) && enable_shadow_vmcs)copy_shadow_to_vmcs12 (vmx); - }/* * The nested entry process starts with enforcing various prerequisites arch/x86/kvm/vmx/vmx.h +4
-1@@ -205,8 +205,11 @@ struct nested_vmx {#ifdef CONFIG_KVM_HYPERV gpa_t hv_evmcs_vmptr; -struct kvm_host_map hv_evmcs_map; +u32 hv_clean_fields; +bool hv_msr_bitmap; +bool hv_flush_hypercall;struct hv_enlightened_vmcs *hv_evmcs; +struct kvm_host_map hv_evmcs_map;#endif };
Signature · 1 line
Vitaly
PATCH v4 7/10 KVM: nVMX: Replace evmcs kvm_host_map with pfncache
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Replace the eVMCS kvm_host_map with a gfn_to_pfn_cache to properly handle memslot changes and unify with other pfncaches in nVMX.
The change introduces proper locking/unlocking semantics for eVMCS access through nested_lock_evmcs() and nested_unlock_evmcs() helpers.
--- arch/x86/kvm/vmx/hyperv.h | 21 +++---- arch/x86/kvm/vmx/nested.c | 115 ++++++++++++++++++++++++++------------ arch/x86/kvm/vmx/vmx.h | 3 +- 3 files changed, 90 insertions(+), 49 deletions(-)
arch/x86/kvm/vmx/hyperv.h +11 -10
@@ -37,11 +37,6 @@ static inline bool nested_vmx_is_evmptr12_set(struct vcpu_vmx *vmx)return evmptr_is_set (vmx->nested.hv_evmcs_vmptr); } -static inline struct hv_enlightened_vmcs *nested_vmx_evmcs (struct vcpu_vmx *vmx) -{ -return vmx->nested.hv_evmcs; -} -static inline bool guest_cpu_cap_has_evmcs (struct kvm_vcpu *vcpu) {/*
@@ -70,6 +65,8 @@ void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *int nested_evmcs_check_controls (struct vmcs12 *vmcs12);bool nested_evmcs_l2_tlb_flush_enabled (struct kvm_vcpu *vcpu);void vmx_hv_inject_synthetic_vmexit_post_tlb_flush (struct kvm_vcpu *vcpu); +struct hv_enlightened_vmcs *nested_lock_evmcs (struct vcpu_vmx *vmx); +void nested_unlock_evmcs (struct vcpu_vmx *vmx);#else static inline bool evmptr_is_valid (u64 evmptr) {
@@ -91,11 +88,6 @@ static inline bool nested_vmx_is_evmptr12_set(struct vcpu_vmx *vmx)return false; } -static inline struct hv_enlightened_vmcs *nested_vmx_evmcs (struct vcpu_vmx *vmx) -{ -return NULL ; -} -static inline u32 nested_evmcs_clean_fields (struct vcpu_vmx *vmx) {return 0 ;
@@ -105,6 +97,15 @@ static inline bool nested_evmcs_msr_bitmap(struct vcpu_vmx *vmx) {return false; } + +static inline struct hv_enlightened_vmcs *nested_lock_evmcs (struct vcpu_vmx *vmx) +{ +return NULL ; +} + +static inline void nested_unlock_evmcs (struct vcpu_vmx *vmx) +{ +}#endif #endif /* __KVM_X86_VMX_HYPERV_H */
arch/x86/kvm/vmx/nested.c +78 -37
253 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +1 -2
@@ -208,8 +208,7 @@ struct nested_vmx {u32 hv_clean_fields;bool hv_msr_bitmap;bool hv_flush_hypercall; -struct hv_enlightened_vmcs *hv_evmcs; -struct kvm_host_map hv_evmcs_map; +struct gfn_to_pfn_cache hv_evmcs_cache;#endif };
Signature · 1 line
2.43.0
PATCH v4 8/10 KVM: x86: Add nested context management
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Add infrastructure to persist nested virtualization state when L2 vCPUs are switched on an L1 vCPU or migrated between L1 vCPUs.
The nested context table uses a hash table for fast lookup by nested control block GPA (VMPTR for VMX, VMCB for SVM) and maintains a free list for context management.
The kvm_nested_context_load() function searches for a context indexed by the target GPA; if not found, it allocates a new context up to the configured maximum. If at capacity, it recycles the oldest context from the free list.
The oversubscription is hardcoded to support up to 8 L2 vCPUs per L1 vCPU.
The kvm_nested_context_clear() function moves the context to the free list while keeping it in the hash table for potential reuse.
This allows nested hypervisors to multiplex multiple L2 vCPUs on L1 vCPUs without losing cached nested state, significantly improving performance for workloads with frequent L2 context switches.
This patch adds the basic infrastructure. Subsequent patches will add the nested VMX and SVM specific support to populate and utilize the cached nested state.
--- arch/x86/include/asm/kvm_host.h | 31 +++++ arch/x86/include/uapi/asm/kvm.h | 2 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/nested.c | 199 ++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 5 +- 5 files changed, 237 insertions(+), 2 deletions(-) create mode 100644 arch/x86/kvm/nested.c
arch/x86/include/asm/kvm_host.h +31 -0
@@ -1376,6 +1376,28 @@ enum kvm_mmu_type { KVM_NR_MMU_TYPES, }; +struct kvm_nested_context { +gpa_t gpa; +struct hlist_node hnode; +struct list_head lru_link; +struct kvm_vcpu *vcpu; +}; + +struct kvm_nested_context_table { +spinlock_t lock; +u32 count; +struct list_head lru_list; + DECLARE_HASHTABLE(hash, 8); +}; + +void kvm_nested_context_clear (struct kvm_vcpu *vcpu,gpa_t gpa); +struct kvm_nested_context *kvm_nested_context_load ( +struct kvm_vcpu *vcpu, +gpa_t gpa); + +int kvm_nested_context_table_init (struct kvm *kvm); +void kvm_nested_context_table_destroy (struct kvm *kvm); +struct kvm_arch {unsigned long n_used_mmu_pages;unsigned long n_requested_mmu_pages;
@@ -1613,6 +1635,9 @@ struct kvm_arch {* current VM. */ int cpu_dirty_log_size; + +/* Cache for nested contexts */ +struct kvm_nested_context_table *nested_context_table; };struct kvm_vm_stat {
@@ -1635,6 +1660,8 @@ struct kvm_vm_stat {u64 nx_lpage_splits;u64 max_mmu_page_hash_collisions;u64 max_mmu_rmap_size; +u64 nested_context_recycle; +u64 nested_context_reuse; };struct kvm_vcpu_stat {
@@ -1963,6 +1990,10 @@ struct kvm_x86_nested_ops { uint16_t *vmcs_version);uint16_t (*get_evmcs_version)(struct kvm_vcpu *vcpu);void (*hv_inject_synthetic_vmexit_post_tlb_flush)(struct kvm_vcpu *vcpu); + +struct kvm_nested_context *(*alloc_context)(struct kvm_vcpu *vcpu); +void (*free_context)(struct kvm_nested_context *ctx); +void (*reset_context)(struct kvm_nested_context *ctx); };struct kvm_x86_init_ops {
arch/x86/include/uapi/asm/kvm.h +2 -0
@@ -1043,4 +1043,6 @@ struct kvm_tdx_init_mem_region { __u64 nr_pages; }; +#define KVM_NESTED_OVERSUB_RATIO 8 +#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/Makefile +1 -1
@@ -6,7 +6,7 @@ ccflags-$(CONFIG_KVM_WERROR) += -Werror include $(srctree)/virt/kvm/Makefile.kvm kvm-y += x86.o emulate.o irq.o lapic.o cpuid.o pmu.o mtrr.o \ - debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o + debugfs.o nested.o mmu/mmu.o mmu/page_track.o mmu/spte.o kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o
arch/x86/kvm/nested.c new file +199 -0
199 lines not on this page · raw message
arch/x86/kvm/x86.c +4 -1
@@ -251,7 +251,9 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {STATS_DESC_ICOUNTER (VM , pages_1g),STATS_DESC_ICOUNTER (VM , nx_lpage_splits),STATS_DESC_PCOUNTER (VM , max_mmu_rmap_size), -STATS_DESC_PCOUNTER (VM , max_mmu_page_hash_collisions) +STATS_DESC_PCOUNTER (VM , max_mmu_page_hash_collisions), +STATS_DESC_COUNTER (VM , nested_context_recycle), +STATS_DESC_COUNTER (VM , nested_context_reuse) }; const struct kvm_stats_header kvm_vm_stats_header = {
@@ -13350,6 +13352,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm)kvm_page_track_cleanup (kvm);kvm_xen_destroy_vm (kvm);kvm_hv_destroy_vm (kvm); +kvm_nested_context_table_destroy (kvm);kvm_x86_call (vm_destroy)(kvm); }
Signature · 1 line
2.43.0
Sean Christopherson
maintainerRe: [PATCH v4 08/10] KVM: x86: Add nested context management
Quoted 3 lines · Fred Griffoul
From: Fred Griffoul <[email protected]>
Add infrastructure to persist nested virtualization state when L2 vCPUs
Please be more transparent with what exactly is being persisted.
Quoted 25 lines
are switched on an L1 vCPU or migrated between L1 vCPUs.
The nested context table uses a hash table for fast lookup by nested control block GPA (VMPTR for VMX, VMCB for SVM) and maintains a free list for context management.
The
kvm_nested_context_load()function searches for a context indexed by the target GPA; if not found, it allocates a new context up to the configured maximum. If at capacity, it recycles the oldest context from the free list.The oversubscription is hardcoded to support up to 8 L2 vCPUs per L1 vCPU.
The
kvm_nested_context_clear()function moves the context to the free list while keeping it in the hash table for potential reuse.This allows nested hypervisors to multiplex multiple L2 vCPUs on L1 vCPUs without losing cached nested state, significantly improving performance for workloads with frequent L2 context switches.
This patch adds the basic infrastructure. Subsequent patches will add the nested VMX and SVM specific support to populate and utilize the cached nested state.
Quoted 8 lines
Signed-off-by Fred Griffoul <[email protected]>--- arch/x86/include/asm/kvm_host.h | 31 +++++ arch/x86/include/uapi/asm/kvm.h | 2 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/nested.c | 199 ++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 5 +- 5 files changed, 237 insertions(+), 2 deletions(-)
Please provide concrete performance numbers. They need to be isolated from the switch to gpcs, and need to show how much benefit is provided for a per-VM hash table vs. (much) simpler approaches, e.g. versus a stupid simple per-vCPU LRU cache, a la KVM's pgd caching.
There also needs to be an analysis of the downsides of the performance gains. If I'm putting the pieces together correctly, quoting a snippet from the cover letter, the performance benefits come from:
The pfncache infrastructure maintains persistent mappings as long as the page GPA does not change, eliminating the memremap/memunmap overhead on every VM entry/exit cycle.
Which means that this caching effectively eliminates the security value added by removing memory from the kernel's direct map. If, in the long term, we're collectively moving towards guest_memfd (for setups that don't want all of the overcommit goodness provided by mm/), then the performance provided by this approach is directly at odds with the efforts to remove guest_memfd memory from the direct map for added security.
E.g. if the ratio of L2:L1 contexts is pushed high enough, it would be possible to have the majority of guest memory mapped into the host kernel.
That then raises the question of whether or not we are optimizing the right thing. E.g. if we can somehow make map+unmap blazing fast for "all" real world usage that matters, then maybe we don't need this type of caching.
In general, this needs a _lot_ more justification on the design decisions. A lot, a lot, a _lot_ more. This is too much code and complexity for me to even start reviewing without hard data.
PATCH v4 9/10 KVM: nVMX: Use nested context for pfncache persistence
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Extend the nested context infrastructure to preserve gfn_to_pfn_cache objects for nested VMX using kvm_nested_context_load() and kvm_nested_context_clear() functions.
The VMX nested context stores gfn_to_pfn_cache structs for:
- MSR permission bitmaps
- APIC access page
- Virtual APIC page
- Posted interrupt descriptor
- Enlightened VMCS
For traditional nested VMX, those pfn caches are loaded upon 'vmptrld' instruction emulation and the context is cleared upon 'vmclear'. This follows the normal L2 vCPU migration sequence of 'vmclear/vmptrld/vmlaunch'.
For enlightened VMCS (eVMCS) support, both functions are called when detecting a change in the eVMCS GPA, ensuring proper context management for Hyper-V nested scenarios.
By preserving the gfn_to_pfn_cache objects across L2 context switches, we avoid costly cache refresh operations, significantly improving nested virtualization performance for workloads with frequent L2 vCPU multiplexing on an L1 vCPU or L2 vCPUs migrations between L1 vCPUs.
--- arch/x86/kvm/vmx/nested.c | 176 ++++++++++++++++++++++++++++++-------- arch/x86/kvm/vmx/vmx.c | 8 ++ arch/x86/kvm/vmx/vmx.h | 10 +-- include/linux/kvm_host.h | 2 +- 4 files changed, 152 insertions(+), 44 deletions(-)
arch/x86/kvm/vmx/nested.c +138 -38
@@ -227,6 +227,93 @@ static void vmx_disable_shadow_vmcs(struct vcpu_vmx *vmx) vmx->nested.need_vmcs12_to_shadow_sync = false; } +struct vmx_nested_context { +struct kvm_nested_context base; +struct gfn_to_pfn_cache msr_bitmap_cache; +struct gfn_to_pfn_cache apic_access_page_cache; +struct gfn_to_pfn_cache virtual_apic_cache; +struct gfn_to_pfn_cache pi_desc_cache; +#ifdef CONFIG_KVM_HYPERV +struct gfn_to_pfn_cache evmcs_cache; +#endif +}; + +static inline struct vmx_nested_context *to_vmx_nested_context ( +struct kvm_nested_context *base) +{ + return base ? container_of(base, struct vmx_nested_context, base) : NULL; +} + +static struct kvm_nested_context *vmx_nested_context_alloc (struct kvm_vcpu *vcpu) +{ +struct vmx_nested_context *ctx; + + ctx =kzalloc (sizeof (*ctx),GFP_KERNEL_ACCOUNT ); +if (!ctx) +return NULL ; + +kvm_gpc_init (&ctx->msr_bitmap_cache, vcpu->kvm); +kvm_gpc_init_for_vcpu (&ctx->apic_access_page_cache, vcpu); +kvm_gpc_init_for_vcpu (&ctx->virtual_apic_cache, vcpu); +kvm_gpc_init_for_vcpu (&ctx->pi_desc_cache, vcpu); +#ifdef CONFIG_KVM_HYPERV +kvm_gpc_init (&ctx->evmcs_cache, vcpu->kvm); +#endif +return &ctx->base; +} + +static void vmx_nested_context_reset (struct kvm_nested_context *base) +{ +/* +* Skip pfncache reinitialization: active ones will be refreshed on +* access. +*/ +} + +static void vmx_nested_context_free (struct kvm_nested_context *base) +{ +struct vmx_nested_context *ctx =to_vmx_nested_context (base); + +kvm_gpc_deactivate (&ctx->pi_desc_cache); +kvm_gpc_deactivate (&ctx->virtual_apic_cache); +kvm_gpc_deactivate (&ctx->apic_access_page_cache); +kvm_gpc_deactivate (&ctx->msr_bitmap_cache); +#ifdef CONFIG_KVM_HYPERV +kvm_gpc_deactivate (&ctx->evmcs_cache); +#endif +kfree (ctx); +} + +static void vmx_nested_context_load (struct vcpu_vmx *vmx,gpa_t vmptr) +{ +struct vmx_nested_context *ctx; + + ctx =to_vmx_nested_context (kvm_nested_context_load (&vmx->vcpu, vmptr)); +if (!ctx) { +/* +* The cache could not be allocated. In the unlikely case of no +* available memory, an error will be returned to L1 when +* mapping the vmcs12 pages. More likely the current pfncaches +* will be reused (and refreshed since their GPAs do not +* match). +*/ +return ; + } + + vmx->nested.msr_bitmap_cache = &ctx->msr_bitmap_cache; + vmx->nested.apic_access_page_cache = &ctx->apic_access_page_cache; + vmx->nested.virtual_apic_cache = &ctx->virtual_apic_cache; + vmx->nested.pi_desc_cache = &ctx->pi_desc_cache; +#ifdef CONFIG_KVM_HYPERV + vmx->nested.hv_evmcs_cache = &ctx->evmcs_cache; +#endif +} + +static void vmx_nested_context_clear (struct vcpu_vmx *vmx,gpa_t vmptr) +{ +kvm_nested_context_clear (&vmx->vcpu, vmptr); +} +static inline void nested_release_evmcs (struct kvm_vcpu *vcpu) {#ifdef CONFIG_KVM_HYPERV
@@ -320,14 +407,23 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, struct loaded_vmcs *vmcs)/* * Maps a single guest page starting at @gpa and lock the cache for access. */ -static int nested_gpc_lock (struct gfn_to_pfn_cache *gpc,gpa_t gpa) +static int nested_gpc_lock (struct kvm_vcpu *vcpu,struct gfn_to_pfn_cache *gpc, +gpa_t gpa) {int err;if (!PAGE_ALIGNED (gpa))return -EINVAL ; + +if (WARN_ON_ONCE (!gpc)) +return -ENOENT ; retry:read_lock (&gpc->lock); + +/* Update vcpu if gpc was recycled */ +if (gpc->vcpu && gpc->vcpu != vcpu) + gpc->vcpu = vcpu; +if (!kvm_gpc_check (gpc,PAGE_SIZE ) || (gpc->gpa != gpa)) {read_unlock (&gpc->lock); err =kvm_gpc_activate (gpc, gpa,PAGE_SIZE );
@@ -355,11 +451,12 @@ static void nested_gpc_unlock(struct gfn_to_pfn_cache *gpc)read_unlock (&gpc->lock); } -static int nested_gpc_hpa (struct gfn_to_pfn_cache *gpc,gpa_t gpa,hpa_t *hpa) +static int nested_gpc_hpa (struct kvm_vcpu *vcpu,struct gfn_to_pfn_cache *gpc, +gpa_t gpa,hpa_t *hpa) {int err; - err =nested_gpc_lock (gpc, gpa); + err =nested_gpc_lock (vcpu, gpc, gpa);if (err)return err;
@@ -427,6 +524,15 @@ static void nested_unlock_vapic(struct vcpu_vmx *vmx)nested_gpc_unlock (vmx->nested.virtual_apic_cache); } +static u64 nested_vmptr (struct vcpu_vmx *vmx) +{ +#ifdef CONFIG_KVM_HYPERV +if (nested_vmx_is_evmptr12_valid (vmx)) +return vmx->nested.hv_evmcs_vmptr; +#endif +return vmx->nested.current_vmptr; +} +/* * Free whatever needs to be freed from vmx->nested when L1 goes down, or * just stops using VMX.
@@ -443,18 +549,12 @@ static void free_nested(struct kvm_vcpu *vcpu)kvm_clear_request (KVM_REQ_GET_NESTED_STATE_PAGES , vcpu); +vmx_nested_context_clear (vmx,nested_vmptr (vmx)); + vmx->nested.vmxon = false; vmx->nested.smm.vmxon = false; vmx->nested.vmxon_ptr =INVALID_GPA ; -kvm_gpc_deactivate (&vmx->nested.pi_desc_cache); -kvm_gpc_deactivate (&vmx->nested.virtual_apic_cache); -kvm_gpc_deactivate (&vmx->nested.apic_access_page_cache); -kvm_gpc_deactivate (&vmx->nested.msr_bitmap_cache); -#ifdef CONFIG_KVM_HYPERV -kvm_gpc_deactivate (&vmx->nested.hv_evmcs_cache); -#endif -free_vpid (vmx->nested.vpid02); vmx->nested.posted_intr_nv =-1 ; vmx->nested.current_vmptr =INVALID_GPA ;
@@ -760,8 +860,8 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu,return true; } - gpc = &vmx->nested.msr_bitmap_cache; -if (nested_gpc_lock (gpc, vmcs12->msr_bitmap)) + gpc = vmx->nested.msr_bitmap_cache; +if (nested_gpc_lock (vcpu, gpc, vmcs12->msr_bitmap))return false; msr_bitmap_l1 = (unsigned long *)gpc->khva;
@@ -2249,17 +2349,22 @@ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld(return EVMPTRLD_DISABLED ; } - gpc = &vmx->nested.hv_evmcs_cache; -if (nested_gpc_lock (gpc, evmcs_gpa)) { +if (evmcs_gpa != vmx->nested.hv_evmcs_vmptr) { +vmx_nested_context_clear (vmx, vmx->nested.hv_evmcs_vmptr); +vmx_nested_context_load (vmx, evmcs_gpa); + evmcs_gpa_changed = true; + } + + gpc = vmx->nested.hv_evmcs_cache; +if (nested_gpc_lock (vcpu, gpc, evmcs_gpa)) {nested_release_evmcs (vcpu);return EVMPTRLD_ERROR ; } evmcs = gpc->khva; -if (unlikely (evmcs_gpa != vmx->nested.hv_evmcs_vmptr)) { +if (evmcs_gpa_changed) { vmx->nested.current_vmptr =INVALID_GPA ; -nested_release_evmcs (vcpu);/*
@@ -2293,7 +2398,6 @@ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld( vmx->nested.hv_evmcs_vmptr = evmcs_gpa; - evmcs_gpa_changed = true;/* * Unlike normal vmcs12, enlightened vmcs12 is not fully * reloaded from guest's memory (read only fields, fields not
@@ -3557,9 +3661,9 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)if (nested_cpu_has2 (vmcs12,SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES )) { - gpc = &vmx->nested.apic_access_page_cache; + gpc = vmx->nested.apic_access_page_cache; -if (!nested_gpc_hpa (gpc, vmcs12->apic_access_addr, &hpa)) { +if (!nested_gpc_hpa (vcpu, gpc, vmcs12->apic_access_addr, &hpa)) {vmcs_write64 (APIC_ACCESS_ADDR , hpa); }else { pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\n",
@@ -3573,9 +3677,9 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) }if (nested_cpu_has (vmcs12,CPU_BASED_TPR_SHADOW )) { - gpc = &vmx->nested.virtual_apic_cache; + gpc = vmx->nested.virtual_apic_cache; -if (!nested_gpc_hpa (gpc, vmcs12->virtual_apic_page_addr, &hpa)) { +if (!nested_gpc_hpa (vcpu, gpc, vmcs12->virtual_apic_page_addr, &hpa)) {vmcs_write64 (VIRTUAL_APIC_PAGE_ADDR , hpa); } else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) && nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) &&
@@ -3599,9 +3703,9 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) }if (nested_cpu_has_posted_intr (vmcs12)) { - gpc = &vmx->nested.pi_desc_cache; + gpc = vmx->nested.pi_desc_cache; -if (!nested_gpc_hpa (gpc, vmcs12->posted_intr_desc_addr &PAGE_MASK , &hpa)) { +if (!nested_gpc_hpa (vcpu, gpc, vmcs12->posted_intr_desc_addr &PAGE_MASK , &hpa)) { vmx->nested.pi_desc_offset =offset_in_page (vmcs12->posted_intr_desc_addr);vmcs_write64 (POSTED_INTR_DESC_ADDR , hpa +offset_in_page (vmcs12->posted_intr_desc_addr));
@@ -3659,9 +3763,9 @@ static bool vmx_is_nested_state_invalid(struct kvm_vcpu *vcpu)* locks. Since kvm_gpc_invalid() doesn't verify gpc memslot * generation, we can also skip acquiring the srcu lock. */ -return kvm_gpc_invalid (&vmx->nested.apic_access_page_cache) || -kvm_gpc_invalid (&vmx->nested.virtual_apic_cache) || -kvm_gpc_invalid (&vmx->nested.pi_desc_cache); +return kvm_gpc_invalid (vmx->nested.apic_access_page_cache) || +kvm_gpc_invalid (vmx->nested.virtual_apic_cache) || +kvm_gpc_invalid (vmx->nested.pi_desc_cache); }static int nested_vmx_write_pml_buffer (struct kvm_vcpu *vcpu,gpa_t gpa)
@@ -4129,7 +4233,6 @@ static void vmcs12_save_pending_event(struct kvm_vcpu *vcpu, } } -void nested_mark_vmcs12_pages_dirty (struct kvm_vcpu *vcpu) {struct vmcs12 *vmcs12 =get_vmcs12 (vcpu);
@@ -5608,16 +5711,6 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) HRTIMER_MODE_ABS_PINNED); vmx->nested.vpid02 =allocate_vpid (); - -kvm_gpc_init (&vmx->nested.msr_bitmap_cache, vcpu->kvm); - -kvm_gpc_init_for_vcpu (&vmx->nested.apic_access_page_cache, vcpu); -kvm_gpc_init_for_vcpu (&vmx->nested.virtual_apic_cache, vcpu); -kvm_gpc_init_for_vcpu (&vmx->nested.pi_desc_cache, vcpu); - -#ifdef CONFIG_KVM_HYPERV -kvm_gpc_init (&vmx->nested.hv_evmcs_cache, vcpu->kvm); -#endif vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true;
@@ -5813,6 +5906,8 @@ static int handle_vmclear(struct kvm_vcpu *vcpu) &zero, sizeof(zero)); } +vmx_nested_context_clear (vmx, vmptr); +return nested_vmx_succeed (vcpu); }
@@ -6057,6 +6152,8 @@ static void set_current_vmptr(struct vcpu_vmx *vmx, gpa_t vmptr) } vmx->nested.dirty_vmcs12 = true; vmx->nested.force_msr_bitmap_recalc = true; + +vmx_nested_context_load (vmx, vmptr); }/* Emulate the VMPTRLD instruction */
@@ -7654,4 +7751,7 @@ struct kvm_x86_nested_ops vmx_nested_ops = { .get_evmcs_version = nested_get_evmcs_version, .hv_inject_synthetic_vmexit_post_tlb_flush = vmx_hv_inject_synthetic_vmexit_post_tlb_flush, #endif + .alloc_context = vmx_nested_context_alloc, + .free_context = vmx_nested_context_free, + .reset_context = vmx_nested_context_reset, };
arch/x86/kvm/vmx/vmx.c +8 -0
@@ -7709,6 +7709,14 @@ int vmx_vm_init(struct kvm *kvm)if (enable_pml) kvm->arch.cpu_dirty_log_size =PML_LOG_NR_ENTRIES ; + +if (nested) { +int err; + + err =kvm_nested_context_table_init (kvm); +if (err) +return err; + }return 0 ; }
arch/x86/kvm/vmx/vmx.h +5 -5
@@ -152,15 +152,15 @@ struct nested_vmx {struct loaded_vmcs vmcs02; -struct gfn_to_pfn_cache msr_bitmap_cache; +struct gfn_to_pfn_cache *msr_bitmap_cache;/* * Guest pages referred to in the vmcs02 with host-physical * pointers, so we must keep them pinned while L2 runs. */ -struct gfn_to_pfn_cache apic_access_page_cache; -struct gfn_to_pfn_cache virtual_apic_cache; -struct gfn_to_pfn_cache pi_desc_cache; +struct gfn_to_pfn_cache *apic_access_page_cache; +struct gfn_to_pfn_cache *virtual_apic_cache; +struct gfn_to_pfn_cache *pi_desc_cache;u64 pi_desc_offset;bool pi_pending;
@@ -208,7 +208,7 @@ struct nested_vmx {u32 hv_clean_fields;bool hv_msr_bitmap;bool hv_flush_hypercall; -struct gfn_to_pfn_cache hv_evmcs_cache; +struct gfn_to_pfn_cache *hv_evmcs_cache;#endif };
include/linux/kvm_host.h +1 -1
@@ -1543,7 +1543,7 @@ static inline bool kvm_gpc_is_hva_active(struct gfn_to_pfn_cache *gpc)static inline bool kvm_gpc_invalid (struct gfn_to_pfn_cache *gpc) { -return gpc->active && !gpc->valid; +return gpc && gpc->active && !gpc->valid; }void kvm_sigset_activate (struct kvm_vcpu *vcpu);
Signature · 1 line
2.43.0
PATCH v4 10/10 KVM: selftests: Add L2 vcpu context switch test
In reply to Fred GriffoulFrom: Fred Griffoul <[email protected]>
Add selftest to validate nested VMX context switching between multiple L2 vCPUs running on the same L1 vCPU. The test exercises both direct VMX interface (using vmptrld/vmclear operations) and enlightened VMCS (eVMCS) interface for Hyper-V nested scenarios.
The test creates multiple VMCS structures and switches between them to verify that the nested_context kvm counters are correct, according to the number of L2 vCPUs and the number of switches.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/vmx_l2_switch_test.c | 416 ++++++++++++++++++ 2 files changed, 417 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/vmx_l2_switch_test.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -140,6 +140,7 @@ TEST_GEN_PROGS_x86 += x86/triple_fault_event_test TEST_GEN_PROGS_x86 += x86/recalc_apic_map_test TEST_GEN_PROGS_x86 += x86/aperfmperf_test TEST_GEN_PROGS_x86 += x86/vmx_apic_update_test +TEST_GEN_PROGS_x86 += x86/vmx_l2_switch_test TEST_GEN_PROGS_x86 += access_tracking_perf_test TEST_GEN_PROGS_x86 += coalesced_io_test TEST_GEN_PROGS_x86 += dirty_log_perf_test
tools/testing/selftests/kvm/x86/vmx_l2_switch_test.c new file +416 -0
416 lines not on this page · raw message
Signature · 1 line
2.43.0
Sean Christopherson
maintainerQuoted 4 lines · Fred Griffoul
First, the current approach is missing proper invalidation handling in critical scenarios. Enlightened VMCS (eVMCS) pages can become stale when memslots are modified, as there is no mechanism to invalidate the cached mappings.
This is a non-issue. Modifying memslots while vCPUs are active will cause problems, period. There is no magic on earth that will prevent that.
Quoted 4 lines
Similarly, APIC access and virtual APIC pages can be migrated by the host, but without proper notification through
mmu_notifiercallbacks, the mappings become invalid and can lead to incorrect behavior.
No, they can't, at least not for unmanaged memory. For kernel-managed memory, KVM obtains a long-term pin, which prevents the host from migrating the page.
Quoted 2 lines · Fred Griffoul
This patch series introduces performance optimizations for nested VMX workloads with "unmanaged" guest memory.
I've reworked the first half of this on top of the GPC SRCU conversion¹, with the guest-mode pinning done as Sean prescribed²:
Quoted 4 lines
IMO, the least awful way to deal with this is to tag
KVM_REQ_GET_NESTED_STATE_PAGESwithKVM_REQUEST_WAIT|KVM_REQUEST_NO_WAKEUP, and then pass the specific request that needs to be made when invalidating a vCPU-mapped gpc as part of__kvm_gpc_init()(though maybe call itvcpu_gpc_init()?).
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/nvmx-gpc
David Woodhouse (2):
KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor)
KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates
Fred Griffoul (3):
KVM: nVMX: Implement cache for L1 MSR bitmap
KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages
KVM: selftests: Add nested VMX APIC cache invalidation test
The caller of kvm_gpc_init_for_vcpu() gives the specific KVM_REQ to use. Invalidation raises that KVM_REQ and forces the vCPU out of guest mode with a KVM_REQ_OUTSIDE_GUEST_MODE. Both kvm_gpc_invalid() and is_nested_state_invalid() go away, along with the entirety of your patch 3. The pending request *is* the thing that signals invalidity, and the vCPU entry path checks it after setting IN_GUEST_MODE.
Testing and review found (and fixed) some issues which are also present in your v4 series...
• vmx_has_nested_events() can't refresh: it's reachable from the kvm_vcpu_block() loop under set_current_state(TASK_INTERRUPTIBLE), and a refresh sleeps. Claim a pending event instead and let the bounce through vcpu_run do the refresh.
• vcpu_enter_guest() consumes KVM_REQ_GET_NESTED_STATE_PAGES *before* calling the handler. If nested_get_vmcs12_pages() then fails (even the load_pdptrs() early exit) and userspace re-runs the vCPU, it re-enters L2 with the stale address still latched and no request pending. With no page reference held, that's use-after-free. Any failure has to re-post the request.
• Stripping KVM_REQUEST_WAIT for !may_block is racy:
process_mrelease() reaps while vCPUs still run. The wait is safe anyway (it spins, never sleeps), so just don't strip it. (Not to mention the fact that the prohibition on sleeping is bogus anyway, qv³).
• A memslot move under a running L2 never re-resolves the latched addresses at all (true of the kvm_host_map code too), so the CPU keeps writing guest-influenced vAPIC/PI state into a page the VMM thinks it took back. Arguably userspace which does this gets what it deserves, but posting the request from kvm_arch_memslots_updated() is cheap and makes it converge the same way every other stale-generation user does.
Your selftest is in the branch (thanks — its madvise mode caught real issues in soak), adjusted for the tdp_map() API churn.
I haven't touched the latter half of your series; didn't either Sean or Vitaly suggest it should be a separate series? If not, I'm suggesting it now :)
Kicking off some tests overnight...
¹ https://lore.kernel.org/all/[email protected]/
² https://lore.kernel.org/all/[email protected]/
³ https://lore.kernel.org/all/[email protected]/
Quoted 4 lines · David Woodhouse
I've reworked the first half of this on top of the GPC SRCU conversion, with the guest-mode pinning done as Sean prescribed:
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/nvmx-gpc
I did the nested SVM equivalent too, now on top of the same tree:
https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/nsvm-gpc
(Those kvm_nested_gpc_lock() helpers have nothing to do with nesting and can be renamed and hoisted to somewhere more generic when I go back and clean it all up, but I'm keeping it incremental for now.)
SVM doesn't use L1 pages directly from the guest VMCB02 (the merged MSRPM and the IOPM are kernel pages). It's only ever direct access from the kernel to gpc->khva.
But that covers the VMCB12 page which is mapped and unmapped twice per L1←→L2 round trip today, and L1's MSR/IO permission bitmaps (kvm_vcpu_read_guest() on each trapped access).
So unlike nVMX, we can even see the win on boring non-PFNMAP memory.
From a cpuid ping-pong between L1 and L2 on EPYC Milan (10k iterations, cycles, measured from L1):
baseline nsvm-gpc reflected exit: p50 47.6k p99 90.2k p50 38.6k p99 81.4k MSR intercept: p50 49.1k p99 90.9k p50 39.9k p99 81.8k
About 9k cycles (~20%) off every emulated vmexit/vmentry. This is a kernel with KASAN+lockdep for correctness torture testing, so the measurement might not be indicative of production performance. I'll do another run on something more representative now it's actually passed the soak tests without catching fire.