KVM: x86: hyper-v: Fine-grained TLB flush + Direct TLB flush feature
Subsystems (6)
Vitaly Kuznetsov
maintainercover KVM: x86: hyper-v: Fine-grained TLB flush + Direct TLB flush feature
Currently, KVM handles HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST{,EX} requests by flushing the whole VPID and this is sub-optimal. This series introduces the required mechanism to make handling of these requests more fine-grained by flushing individual GVAs only (when requested). On this foundation, "Direct Virtual Flush" Hyper-V feature is implemented. The feature allows L0 to handle Hyper-V TLB flush hypercalls directly at L0 without the need to reflect the exit to L1. This has at least two
benefits: reflecting vmexit and the consequent vmenter are avoided + L0 has precise information whether the target vCPU is actually running (and thus requires a kick).
RFC part:
- The series is poorly tested. I've only smoke-tested WS2019 and Win11+WSL2 on VMX and AMD harware to the point that no immediate issues are observed on boot. We certainly need something added to selftests/kvm-unit-tests.
Vitaly Kuznetsov (19):
KVM: x86: hyper-v: Resurrect dedicated KVM_REQ_HV_TLB_FLUSH flag
KVM: x86: hyper-v: Introduce TLB flush ring
KVM: x86: hyper-v: Handle HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST{,EX} calls
gently
KVM: x86: hyper-v: Expose support for extended gva ranges for flush
hypercalls
KVM: x86: Prepare kvm_hv_flush_tlb() to handle L2's GPAs
KVM: x86: hyper-v: Don't use sparse_set_to_vcpu_mask() in
kvm_hv_send_ipi()
KVM: x86: hyper-v: Create a separate ring for Direct TLB flush
KVM: x86: hyper-v: Use preallocated buffer in 'struct kvm_vcpu_hv'
instead of on-stack 'sparse_banks'
KVM: nVMX: Keep track of hv_vm_id/hv_vp_id when eVMCS is in use
KVM: nSVM: Keep track of Hyper-V hv_vm_id/hv_vp_id
KVM: x86: Introduce .post_hv_direct_flush() nested hook
KVM: x86: hyper-v: Introduce kvm_hv_is_tlb_flush_hcall()
KVM: x86: hyper-v: Direct TLB flush
KVM: x86: hyper-v: Introduce fast kvm_hv_direct_tlb_flush_exposed()
check
x86/hyperv: Fix 'struct hv_enlightened_vmcs' definition
KVM: nVMX: hyper-v: Direct TLB flush
KVM: x86: KVM_REQ_TLB_FLUSH_CURRENT is a superset of
KVM_REQ_HV_TLB_FLUSH too
KVM: nSVM: hyper-v: Direct TLB flush
KVM: x86: Expose Hyper-V Direct TLB flush feature
arch/x86/include/asm/hyperv-tlfs.h | 6 +- arch/x86/include/asm/kvm_host.h | 30 +++ arch/x86/kvm/Makefile | 3 +- arch/x86/kvm/hyperv.c | 305 ++++++++++++++++++++++++++--- arch/x86/kvm/hyperv.h | 55 ++++++ arch/x86/kvm/svm/hyperv.c | 18 ++ arch/x86/kvm/svm/hyperv.h | 37 ++++ arch/x86/kvm/svm/nested.c | 25 ++- arch/x86/kvm/trace.h | 21 +- arch/x86/kvm/vmx/evmcs.c | 24 +++ arch/x86/kvm/vmx/evmcs.h | 4 + arch/x86/kvm/vmx/nested.c | 41 +++- arch/x86/kvm/x86.c | 15 +- arch/x86/kvm/x86.h | 1 + 14 files changed, 539 insertions(+), 46 deletions(-) create mode 100644 arch/x86/kvm/svm/hyperv.c
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 1/19 KVM: x86: hyper-v: Resurrect dedicated KVM_REQ_HV_TLB_FLUSH flag
In preparation to implementing fine-grained Hyper-V TLB flush and Direct TLB flush, resurrect dedicated KVM_REQ_HV_TLB_FLUSH request bit. As KVM_REQ_TLB_FLUSH_GUEST is a stronger operation, clear KVM_REQ_HV_TLB_FLUSH request in kvm_service_local_tlb_flush_requests() when KVM_REQ_TLB_FLUSH_GUEST was also requested.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/hyperv.c | 4 ++-- arch/x86/kvm/x86.c | 7 ++++++- 3 files changed, 10 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -104,6 +104,8 @@ KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_MMU_FREE_OBSOLETE_ROOTS \ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_HV_TLB_FLUSH \ + KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
arch/x86/kvm/hyperv.c +2 -2
@@ -1890,11 +1890,11 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)* analyze it here, flush TLB regardless of the specified address space. */ if (all_cpus) { -kvm_make_all_cpus_request (kvm,KVM_REQ_TLB_FLUSH_GUEST ); +kvm_make_all_cpus_request (kvm,KVM_REQ_HV_TLB_FLUSH ); }else {sparse_set_to_vcpu_mask (kvm, sparse_banks, valid_bank_mask, vcpu_mask); -kvm_make_vcpus_request_mask (kvm,KVM_REQ_TLB_FLUSH_GUEST , vcpu_mask); +kvm_make_vcpus_request_mask (kvm,KVM_REQ_HV_TLB_FLUSH , vcpu_mask); } ret_success:
arch/x86/kvm/x86.c +6 -1
@@ -3339,7 +3339,12 @@ void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu))kvm_vcpu_flush_tlb_current (vcpu); -if (kvm_check_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu)) +if (kvm_check_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu)) { +kvm_vcpu_flush_tlb_guest (vcpu); +kvm_clear_request (KVM_REQ_HV_TLB_FLUSH , vcpu); + } + +if (kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu))kvm_vcpu_flush_tlb_guest (vcpu); }EXPORT_SYMBOL_GPL (kvm_service_local_tlb_flush_requests);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 2/19 KVM: x86: hyper-v: Introduce TLB flush ring
In reply to Vitaly KuznetsovTo allow flushing individual GVAs instead of always flushing the whole VPID a per-vCPU structure to pass the requests is needed. Introduce a simple ring write-locked structure to hold two types of entries:
individual GVA (GFN + up to 4095 following GFNs in the lower 12 bits) and 'flush all'.
The queuing rule is: if there's not enough space on the ring to put the request and leave at least 1 entry for 'flush all' - put 'flush all' entry.
The size of the ring is arbitrary set to '16'.
Note, kvm_hv_flush_tlb() only queues 'flush all' entries for now so there's very small functional change but the infrastructure is prepared to handle individual GVA flush requests.
--- arch/x86/include/asm/kvm_host.h | 16 +++++++ arch/x86/kvm/hyperv.c | 74 +++++++++++++++++++++++++++++++++ arch/x86/kvm/hyperv.h | 13 ++++++ arch/x86/kvm/x86.c | 7 ++-- arch/x86/kvm/x86.h | 1 + 5 files changed, 108 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +16 -0
@@ -579,6 +579,20 @@ struct kvm_vcpu_hv_synic {bool dont_zero_synic_pages; }; +#define KVM_HV_TLB_FLUSH_RING_SIZE (16) + +struct kvm_vcpu_hv_tlbflush_entry { +u64 addr; +u64 flush_all:1 ; +u64 pad:63 ; +}; + +struct kvm_vcpu_hv_tlbflush_ring { +int read_idx, write_idx; +spinlock_t write_lock; +struct kvm_vcpu_hv_tlbflush_entry entries[KVM_HV_TLB_FLUSH_RING_SIZE ]; +}; +/* Hyper-V per vcpu emulation context */ struct kvm_vcpu_hv {struct kvm_vcpu *vcpu;
@@ -598,6 +612,8 @@ struct kvm_vcpu_hv {u32 enlightenments_ebx;/* HYPERV_CPUID_ENLIGHTMENT_INFO.EBX */ u32 syndbg_cap_eax;/* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ } cpuid_cache; + +struct kvm_vcpu_hv_tlbflush_ring tlb_flush_ring; };/* Xen HVM per vcpu emulation context */
arch/x86/kvm/hyperv.c +74 -0
108 lines not on this page · raw message
arch/x86/kvm/hyperv.h +13 -0
@@ -147,4 +147,17 @@ int kvm_vm_ioctl_hv_eventfd(struct kvm *kvm, struct kvm_hyperv_eventfd *args);int kvm_get_hv_cpuid (struct kvm_vcpu *vcpu,struct kvm_cpuid2 *cpuid,struct kvm_cpuid_entry2 __user *entries); + +static inline void kvm_hv_vcpu_empty_flush_tlb (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); + +if (!hv_vcpu) +return ; + + hv_vcpu->tlb_flush_ring.read_idx = hv_vcpu->tlb_flush_ring.write_idx; +} +void kvm_hv_vcpu_flush_tlb (struct kvm_vcpu *vcpu); + +#endif
arch/x86/kvm/x86.c +4 -3
@@ -3303,7 +3303,7 @@ static void kvm_vcpu_flush_tlb_all(struct kvm_vcpu *vcpu)static_call (kvm_x86_flush_tlb_all)(vcpu); } -static void kvm_vcpu_flush_tlb_guest (struct kvm_vcpu *vcpu) +void kvm_vcpu_flush_tlb_guest (struct kvm_vcpu *vcpu) { ++vcpu->stat.tlb_flush;
@@ -3341,11 +3341,12 @@ void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu)if (kvm_check_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu)) {kvm_vcpu_flush_tlb_guest (vcpu); -kvm_clear_request (KVM_REQ_HV_TLB_FLUSH , vcpu); +if (kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu)) +kvm_hv_vcpu_empty_flush_tlb (vcpu); }if (kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu)) -kvm_vcpu_flush_tlb_guest (vcpu); +kvm_hv_vcpu_flush_tlb (vcpu); }EXPORT_SYMBOL_GPL (kvm_service_local_tlb_flush_requests);
arch/x86/kvm/x86.h +1 -0
@@ -58,6 +58,7 @@ static inline unsigned int __shrink_ple_window(unsigned int val,#define MSR_IA32_CR_PAT_DEFAULT 0x0007040600070406ULL +void kvm_vcpu_flush_tlb_guest (struct kvm_vcpu *vcpu);void kvm_service_local_tlb_flush_requests (struct kvm_vcpu *vcpu);int kvm_check_nested_events (struct kvm_vcpu *vcpu);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 3/19 KVM: x86: hyper-v: Handle HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST{,EX} calls gently
In reply to Vitaly KuznetsovCurrently, HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST{,EX} calls are handled the exact same way as HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE{,EX}: by flushing the whole VPID and this is sub-optimal. Switch to handling these requests with 'flush_tlb_gva()' hooks instead. Use the newly introduced TLB flush ring to queue the requests.
--- arch/x86/kvm/hyperv.c | 141 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 121 insertions(+), 20 deletions(-)
arch/x86/kvm/hyperv.c +121 -20
@@ -1806,6 +1806,35 @@ static u64 kvm_get_sparse_vp_set(struct kvm *kvm, struct kvm_hv_hcall *hc, var_cnt * sizeof(*sparse_banks)); } +static int kvm_hv_get_tlbflush_entries (struct kvm *kvm,struct kvm_hv_hcall *hc,u64 entries[], +u32 data_offset,int consumed_xmm_halves) +{ +int i; + +if (hc->fast) { +/* +* Each XMM holds two entries, but do not count halves that +* have already been consumed. +*/ +if (hc->rep_cnt > (2 *HV_HYPERCALL_MAX_XMM_REGISTERS - consumed_xmm_halves)) +return -EINVAL ; + +for (i =0 ; i < hc->rep_cnt; i++) { +int j = i + consumed_xmm_halves; + +if (j %2 ) + entries[i] =sse128_hi (hc->xmm[j /2 ]); +else + entries[i] =sse128_lo (hc->xmm[j /2 ]); + } + +return 0 ; + } + +return kvm_read_guest (kvm, hc->ingpa + data_offset, + entries, hc->rep_cnt *sizeof (entries[0 ])); +} +static inline int hv_tlb_flush_ring_free (struct kvm_vcpu_hv *hv_vcpu,int read_idx,int write_idx) {
@@ -1815,12 +1844,14 @@ static inline int hv_tlb_flush_ring_free(struct kvm_vcpu_hv *hv_vcpu,return read_idx - write_idx -1 ; } -static void hv_tlb_flush_ring_enqueue (struct kvm_vcpu *vcpu) +static void hv_tlb_flush_ring_enqueue (struct kvm_vcpu *vcpu,bool flush_all, +u64 *entries,int count) {struct kvm_vcpu_hv_tlbflush_ring *tlb_flush_ring;struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);int ring_free, write_idx, read_idx;unsigned long flags; +int i;if (!hv_vcpu)return ;
@@ -1837,14 +1868,34 @@ static void hv_tlb_flush_ring_enqueue(struct kvm_vcpu *vcpu)if (!ring_free) goto out_unlock; - tlb_flush_ring->entries[write_idx].addr =0 ; - tlb_flush_ring->entries[write_idx].flush_all =1 ;/* -* Advance write index only after filling in the entry to -* synchronize with lockless reader. +* All entries should fit on the ring leaving one free for 'flush all' +* entry in case another request comes in. In case there's not enough +* space, just put 'flush all' entry there. +*/ +if (!count || count >= ring_free -1 || flush_all) { + tlb_flush_ring->entries[write_idx].addr =0 ; + tlb_flush_ring->entries[write_idx].flush_all =1 ; +/* +* Advance write index only after filling in the entry to +* synchronize with lockless reader. +*/ +smp_wmb (); + tlb_flush_ring->write_idx = (write_idx +1 ) %KVM_HV_TLB_FLUSH_RING_SIZE ; + goto out_unlock; + } + +for (i =0 ; i < count; i++) { + tlb_flush_ring->entries[write_idx].addr = entries[i]; + tlb_flush_ring->entries[write_idx].flush_all =0 ; + write_idx = (write_idx +1 ) %KVM_HV_TLB_FLUSH_RING_SIZE ; + } +/* +* Advance write index only after filling in the entry to synchronize +* with lockless reader. */ smp_wmb (); - tlb_flush_ring->write_idx = (write_idx +1 ) %KVM_HV_TLB_FLUSH_RING_SIZE ; + tlb_flush_ring->write_idx = write_idx; out_unlock:spin_unlock_irqrestore (&tlb_flush_ring->write_lock, flags);
@@ -1854,15 +1905,47 @@ void kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv_tlbflush_ring *tlb_flush_ring;struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); - -kvm_vcpu_flush_tlb_guest (vcpu); - -if (!hv_vcpu) +struct kvm_vcpu_hv_tlbflush_entry *entry; +int read_idx, write_idx; +u64 address; +u32 count; +int i, j; + +if (!tdp_enabled || !hv_vcpu) { +kvm_vcpu_flush_tlb_guest (vcpu);return ; + } tlb_flush_ring = &hv_vcpu->tlb_flush_ring; + read_idx =READ_ONCE (tlb_flush_ring->read_idx); + write_idx =READ_ONCE (tlb_flush_ring->write_idx); + +/* Pairs with smp_wmb() in hv_tlb_flush_ring_enqueue() */ +smp_rmb (); - tlb_flush_ring->read_idx = tlb_flush_ring->write_idx; +for (i = read_idx; i != write_idx; i = (i +1 ) %KVM_HV_TLB_FLUSH_RING_SIZE ) { + entry = &tlb_flush_ring->entries[i]; + +if (entry->flush_all) + goto out_flush_all; + +/* +* Lower 12 bits of 'address' encode the number of additional +* pages to flush. +*/ + address = entry->addr &PAGE_MASK ; + count = (entry->addr & ~PAGE_MASK ) +1 ; +for (j =0 ; j < count; j++) +static_call (kvm_x86_flush_tlb_gva)(vcpu, address + j *PAGE_SIZE ); + } + ++vcpu->stat.tlb_flush; + goto out_empty_ring; + +out_flush_all: +kvm_vcpu_flush_tlb_guest (vcpu); + +out_empty_ring: + tlb_flush_ring->read_idx = write_idx; }static u64 kvm_hv_flush_tlb (struct kvm_vcpu *vcpu,struct kvm_hv_hcall *hc)
@@ -1871,12 +1954,13 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)struct hv_tlb_flush_ex flush_ex;struct hv_tlb_flush flush;DECLARE_BITMAP (vcpu_mask,KVM_MAX_VCPUS ); +u64 entries[KVM_HV_TLB_FLUSH_RING_SIZE -2 ];u64 valid_bank_mask;u64 sparse_banks[KVM_HV_MAX_SPARSE_VCPU_SET_BITS ];struct kvm_vcpu *v;unsigned long i; -bool all_cpus; - +bool all_cpus, all_addr; +int data_offset =0 , consumed_xmm_halves =0 ;/* * The Hyper-V TLFS doesn't allow more than 64 sparse banks, e.g. the * valid mask is a u64. Fail the build if KVM's max allowed number of
@@ -1891,10 +1975,12 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) flush.address_space = hc->ingpa; flush.flags = hc->outgpa; flush.processor_mask =sse128_lo (hc->xmm[0 ]); + consumed_xmm_halves =1 ; }else {if (unlikely (kvm_read_guest (kvm, hc->ingpa, &flush,sizeof (flush))))return HV_STATUS_INVALID_HYPERCALL_INPUT ; + data_offset =sizeof (flush); } trace_kvm_hv_flush_tlb(flush.processor_mask,
@@ -1918,10 +2004,12 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) flush_ex.flags = hc->outgpa;memcpy (&flush_ex.hv_vp_set, &hc->xmm[0 ],sizeof (hc->xmm[0 ])); + consumed_xmm_halves =2 ; }else {if (unlikely (kvm_read_guest (kvm, hc->ingpa, &flush_ex,sizeof (flush_ex))))return HV_STATUS_INVALID_HYPERCALL_INPUT ; + data_offset =sizeof (flush_ex); } trace_kvm_hv_flush_tlb_ex(flush_ex.hv_vp_set.valid_bank_mask,
@@ -1937,25 +2025,38 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)return HV_STATUS_INVALID_HYPERCALL_INPUT ;if (all_cpus) - goto do_flush; + goto read_flush_entries;if (!hc->var_cnt) goto ret_success; -if (kvm_get_sparse_vp_set (kvm, hc,2 , sparse_banks, - offsetof(struct hv_tlb_flush_ex , - hv_vp_set.bank_contents))) +if (kvm_get_sparse_vp_set (kvm, hc, consumed_xmm_halves, + sparse_banks, data_offset)) +return HV_STATUS_INVALID_HYPERCALL_INPUT ; + data_offset += hc->var_cnt *sizeof (sparse_banks[0 ]); + consumed_xmm_halves += hc->var_cnt; + } + +read_flush_entries: +if (hc->code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE || + hc->code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX || + hc->rep_cnt > (KVM_HV_TLB_FLUSH_RING_SIZE -2 )) { + all_addr = true; + }else { +if (kvm_hv_get_tlbflush_entries (kvm, hc, entries, + data_offset, consumed_xmm_halves))return HV_STATUS_INVALID_HYPERCALL_INPUT ; + all_addr = false; } -do_flush: +/* * vcpu->arch.cr3 may not be up-to-date for running vCPUs so we can't * analyze it here, flush TLB regardless of the specified address space. */ if (all_cpus) {kvm_for_each_vcpu (i, v, kvm) -hv_tlb_flush_ring_enqueue (v); +hv_tlb_flush_ring_enqueue (v, all_addr, entries, hc->rep_cnt);kvm_make_all_cpus_request (kvm,KVM_REQ_HV_TLB_FLUSH ); }else {
@@ -1965,7 +2066,7 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) v =kvm_get_vcpu (kvm, i);if (!v)continue ; -hv_tlb_flush_ring_enqueue (v); +hv_tlb_flush_ring_enqueue (v, all_addr, entries, hc->rep_cnt); }kvm_make_vcpus_request_mask (kvm,KVM_REQ_HV_TLB_FLUSH , vcpu_mask);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 4/19 KVM: x86: hyper-v: Expose support for extended gva ranges for flush hypercalls
In reply to Vitaly KuznetsovExtended GVA ranges support bit seems to indicate whether lower 12 bits of GVA can be used to specify up to 4095 additional consequent GVAs to flush. This is somewhat described in TLFS.
Previously, KVM was handling HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST{,EX} requests by flushing the whole VPID so technically, extended GVA ranges were already supported. As such requests are handled more gently now, advertizing support for extended ranges starts making sense to reduce the size of TLB flush requests.
--- arch/x86/include/asm/hyperv-tlfs.h | 2 ++ arch/x86/kvm/hyperv.c | 1 + 2 files changed, 3 insertions(+)
arch/x86/include/asm/hyperv-tlfs.h +2 -0
@@ -61,6 +61,8 @@#define HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE BIT(10)/* Support for debug MSRs available */ #define HV_FEATURE_DEBUG_MSRS_AVAILABLE BIT(11) +/* Support for extended gva ranges for flush hypercalls available */ +#define HV_FEATURE_EXT_GVA_RANGES_FLUSH BIT(14)/* * Support for returning hypercall output block via XMM * registers is available
arch/x86/kvm/hyperv.c +1 -0
@@ -2697,6 +2697,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, ent->ebx |=HV_DEBUGGING ; ent->edx |=HV_X64_GUEST_DEBUGGING_AVAILABLE ; ent->edx |=HV_FEATURE_DEBUG_MSRS_AVAILABLE ; + ent->edx |=HV_FEATURE_EXT_GVA_RANGES_FLUSH ;/* * Direct Synthetic timers only make sense with in-kernel
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 5/19 KVM: x86: Prepare kvm_hv_flush_tlb() to handle L2's GPAs
In reply to Vitaly KuznetsovTo handle Direct TLB flush requests from L2, KVM needs to translate the specified L2 GPA to L1 GPA to read hypercall arguments from there.
No fucntional change as KVM doesn't handle VMCALL/VMMCALL from L2 yet.
--- arch/x86/kvm/hyperv.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kvm/hyperv.c +7 -0
@@ -23,6 +23,7 @@#include "ioapic.h" #include "cpuid.h" #include "hyperv.h" +#include "mmu.h" #include "xen.h" #include <linux/cpu.h>
@@ -1969,6 +1970,12 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)*/ BUILD_BUG_ON (KVM_HV_MAX_SPARSE_VCPU_SET_BITS >64 ); +if (!hc->fast &&is_guest_mode (vcpu)) { + hc->ingpa =translate_nested_gpa (vcpu, hc->ingpa,0 ,NULL ); +if (unlikely (hc->ingpa ==UNMAPPED_GVA )) +return HV_STATUS_INVALID_HYPERCALL_INPUT ; + } +if (hc->code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST || hc->code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE ) {if (hc->fast) {
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 6/19 KVM: x86: hyper-v: Don't use sparse_set_to_vcpu_mask() in kvm_hv_send_ipi()
In reply to Vitaly KuznetsovGet rid of on-stack allocation of vcpu_mask and optimize kvm_hv_send_ipi() for a smaller number of vCPUs in the request. When Hyper-V TLB flush is in use, HvSendSyntheticClusterIpi{,Ex} calls are not commonly used to send IPIs to a large number of vCPUs (and are rarely used in general).
Introduce hv_is_vp_in_sparse_set() to directly check if the specified VP_ID is present in sparse vCPU set.
--- arch/x86/kvm/hyperv.c | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-)
arch/x86/kvm/hyperv.c +24 -11
@@ -1760,6 +1760,23 @@ static void sparse_set_to_vcpu_mask(struct kvm *kvm, u64 *sparse_banks, } } +static bool hv_is_vp_in_sparse_set (u32 vp_id,u64 valid_bank_mask,u64 sparse_banks[]) +{ +int bank, sbank =0 ; + +if (!test_bit (vp_id /64 , (unsigned long *)&valid_bank_mask)) +return false; + +for_each_set_bit (bank, (unsigned long *)&valid_bank_mask, +KVM_HV_MAX_SPARSE_VCPU_SET_BITS ) { +if (bank == vp_id /64 ) +break ; + sbank++; + } + +return test_bit (vp_id %64 , (unsigned long *)&sparse_banks[sbank]); +} +struct kvm_hv_hcall {u64 param;u64 ingpa;
@@ -2085,8 +2102,8 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) ((u64 )hc->rep_cnt <<HV_HYPERCALL_REP_COMP_OFFSET ); } -static void kvm_send_ipi_to_many (struct kvm *kvm,u32 vector, -unsigned long *vcpu_bitmap) +static void kvm_hv_send_ipi_to_many (struct kvm *kvm,u32 vector, +u64 *sparse_banks,u64 valid_bank_mask) {struct kvm_lapic_irq irq = { .delivery_mode =APIC_DM_FIXED ,
@@ -2096,7 +2113,10 @@ static void kvm_send_ipi_to_many(struct kvm *kvm, u32 vector,unsigned long i;kvm_for_each_vcpu (i, vcpu, kvm) { -if (vcpu_bitmap && !test_bit (i, vcpu_bitmap)) +if (sparse_banks && + !hv_is_vp_in_sparse_set (kvm_hv_get_vpindex (vcpu), + valid_bank_mask, + sparse_banks))continue ;/* We fail only when APIC is disabled */
@@ -2109,7 +2129,6 @@ static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)struct kvm *kvm = vcpu->kvm;struct hv_send_ipi_ex send_ipi_ex;struct hv_send_ipi send_ipi; -DECLARE_BITMAP (vcpu_mask,KVM_MAX_VCPUS );unsigned long valid_bank_mask;u64 sparse_banks[KVM_HV_MAX_SPARSE_VCPU_SET_BITS ];u32 vector;
@@ -2171,13 +2190,7 @@ static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)if ((vector <HV_IPI_LOW_VECTOR ) || (vector >HV_IPI_HIGH_VECTOR ))return HV_STATUS_INVALID_HYPERCALL_INPUT ; -if (all_cpus) { -kvm_send_ipi_to_many (kvm, vector,NULL ); - }else { -sparse_set_to_vcpu_mask (kvm, sparse_banks, valid_bank_mask, vcpu_mask); - -kvm_send_ipi_to_many (kvm, vector, vcpu_mask); - } +kvm_hv_send_ipi_to_many (kvm, vector, all_cpus ?NULL : sparse_banks, valid_bank_mask); ret_success:return HV_STATUS_SUCCESS ;
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 7/19 KVM: x86: hyper-v: Create a separate ring for Direct TLB flush
In reply to Vitaly KuznetsovTo handle Direct TLB flush requests from L2 KVM needs to use a separate ring from regular Hyper-V TLB flush requests: e.g. when a request to flush something in L2 is made, the target vCPU can transition from L2 to L1, receive a request to flush a GVA for L1 and then try to enter L2 back. The first request needs to be processed then. Similarly, requests to flush GVAs in L1 must wait until L2 exits to L1.
No functional change yet as KVM doesn't handle Direct TLB flush requests from L2 yet.
--- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/hyperv.c | 7 ++++--- arch/x86/kvm/hyperv.h | 17 ++++++++++++++--- 3 files changed, 20 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
@@ -613,7 +613,8 @@ struct kvm_vcpu_hv {u32 syndbg_cap_eax;/* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ } cpuid_cache; -struct kvm_vcpu_hv_tlbflush_ring tlb_flush_ring; +/* Two rings for regular Hyper-V TLB flush and Direct TLB flush */ +struct kvm_vcpu_hv_tlbflush_ring tlb_flush_ring[2 ]; };/* Xen HVM per vcpu emulation context */
arch/x86/kvm/hyperv.c +4 -3
@@ -946,7 +946,8 @@ static int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu) hv_vcpu->vp_index = vcpu->vcpu_idx; -spin_lock_init (&hv_vcpu->tlb_flush_ring.write_lock); +spin_lock_init (&hv_vcpu->tlb_flush_ring[0 ].write_lock); +spin_lock_init (&hv_vcpu->tlb_flush_ring[1 ].write_lock);return 0 ; }
@@ -1874,7 +1875,7 @@ static void hv_tlb_flush_ring_enqueue(struct kvm_vcpu *vcpu, bool flush_all,if (!hv_vcpu)return ; - tlb_flush_ring = &hv_vcpu->tlb_flush_ring; + tlb_flush_ring = &hv_vcpu->tlb_flush_ring[0 ];spin_lock_irqsave (&tlb_flush_ring->write_lock, flags);
@@ -1934,7 +1935,7 @@ void kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)return ; } - tlb_flush_ring = &hv_vcpu->tlb_flush_ring; + tlb_flush_ring =kvm_hv_get_tlb_flush_ring (vcpu); read_idx =READ_ONCE (tlb_flush_ring->read_idx); write_idx =READ_ONCE (tlb_flush_ring->write_idx);
arch/x86/kvm/hyperv.h +14 -3
@@ -22,6 +22,7 @@#define __ARCH_X86_KVM_HYPERV_H__#include <linux/kvm_host.h> +#include "x86.h" /* * The #defines related to the synthetic debugger are required by KDNet, but
@@ -147,15 +148,25 @@ int kvm_vm_ioctl_hv_eventfd(struct kvm *kvm, struct kvm_hyperv_eventfd *args);int kvm_get_hv_cpuid (struct kvm_vcpu *vcpu,struct kvm_cpuid2 *cpuid,struct kvm_cpuid_entry2 __user *entries); +static inline struct kvm_vcpu_hv_tlbflush_ring *kvm_hv_get_tlb_flush_ring (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); + +if (!is_guest_mode (vcpu)) +return &hv_vcpu->tlb_flush_ring[0 ]; + +return &hv_vcpu->tlb_flush_ring[1 ]; +}static inline void kvm_hv_vcpu_empty_flush_tlb (struct kvm_vcpu *vcpu) { -struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +struct kvm_vcpu_hv_tlbflush_ring *tlb_flush_ring; -if (!hv_vcpu) +if (!to_hv_vcpu (vcpu))return ; - hv_vcpu->tlb_flush_ring.read_idx = hv_vcpu->tlb_flush_ring.write_idx; + tlb_flush_ring =kvm_hv_get_tlb_flush_ring (vcpu); + tlb_flush_ring->read_idx = tlb_flush_ring->write_idx; }void kvm_hv_vcpu_flush_tlb (struct kvm_vcpu *vcpu);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 8/19 KVM: x86: hyper-v: Use preallocated buffer in 'struct kvm_vcpu_hv' instead of on-stack 'sparse_banks'
In reply to Vitaly KuznetsovTo make kvm_hv_flush_tlb() ready to handle Direct TLB flush request KVM needs to allow for all 64 sparse vCPU banks regardless of KVM_MAX_VCPUs as L1 may use vCPU overcommit for L2. To avoid growing on-stack allocation, make 'sparse_banks' part of per-vCPU 'struct kvm_vcpu_hv' which is allocated dynamically.
Note: sparse_set_to_vcpu_mask() keeps using on-stack allocation as it won't be used to handle Direct TLB flush requests.
--- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/hyperv.c | 6 ++++-- 2 files changed, 7 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -615,6 +615,9 @@ struct kvm_vcpu_hv {/* Two rings for regular Hyper-V TLB flush and Direct TLB flush */ struct kvm_vcpu_hv_tlbflush_ring tlb_flush_ring[2 ]; + +/* Preallocated buffer for handling hypercalls passing sparse vCPU set */ +u64 sparse_banks[64 ]; };/* Xen HVM per vcpu emulation context */
arch/x86/kvm/hyperv.c +4 -2
@@ -1969,13 +1969,14 @@ void kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)static u64 kvm_hv_flush_tlb (struct kvm_vcpu *vcpu,struct kvm_hv_hcall *hc) { +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +u64 *sparse_banks = hv_vcpu->sparse_banks;struct kvm *kvm = vcpu->kvm;struct hv_tlb_flush_ex flush_ex;struct hv_tlb_flush flush;DECLARE_BITMAP (vcpu_mask,KVM_MAX_VCPUS );u64 entries[KVM_HV_TLB_FLUSH_RING_SIZE -2 ];u64 valid_bank_mask; -u64 sparse_banks[KVM_HV_MAX_SPARSE_VCPU_SET_BITS ];struct kvm_vcpu *v;unsigned long i;bool all_cpus, all_addr;
@@ -2127,11 +2128,12 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,static u64 kvm_hv_send_ipi (struct kvm_vcpu *vcpu,struct kvm_hv_hcall *hc) { +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +u64 *sparse_banks = hv_vcpu->sparse_banks;struct kvm *kvm = vcpu->kvm;struct hv_send_ipi_ex send_ipi_ex;struct hv_send_ipi send_ipi;unsigned long valid_bank_mask; -u64 sparse_banks[KVM_HV_MAX_SPARSE_VCPU_SET_BITS ];u32 vector;bool all_cpus;
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 9/19 KVM: nVMX: Keep track of hv_vm_id/hv_vp_id when eVMCS is in use
In reply to Vitaly KuznetsovTo handle Direct TLB flush requests from L2, KVM needs to keep track of L2's VM_ID/VP_IDs which are set by L1 hypervisor. 'Partition assist page' address is also needed to handle post-flush exit to L1 (upon request).
Assume vm_id/vp_id/partition_assist_page don't change while eVMCS is active.
--- arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/vmx/nested.c | 24 ++++++++++++++++++++---- 2 files changed, 26 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
@@ -618,6 +618,12 @@ struct kvm_vcpu_hv {/* Preallocated buffer for handling hypercalls passing sparse vCPU set */ u64 sparse_banks[64 ]; + +struct { +u64 pa_page_gpa; +u64 vm_id; +u32 vp_id; + } nested; };/* Xen HVM per vcpu emulation context */
arch/x86/kvm/vmx/nested.c +20 -4
@@ -225,6 +225,7 @@ static void vmx_disable_shadow_vmcs(struct vcpu_vmx *vmx)static inline void nested_release_evmcs (struct kvm_vcpu *vcpu) { +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);struct vcpu_vmx *vmx =to_vmx (vcpu);if (evmptr_is_valid (vmx->nested.hv_evmcs_vmptr)) {
@@ -233,6 +234,9 @@ static inline void nested_release_evmcs(struct kvm_vcpu *vcpu) } vmx->nested.hv_evmcs_vmptr =EVMPTR_INVALID ; + hv_vcpu->nested.pa_page_gpa =INVALID_GPA ; + hv_vcpu->nested.vm_id =0 ; + hv_vcpu->nested.vp_id =0 ; } static void vmx_sync_vmcs_host_state(struct vcpu_vmx *vmx,
@@ -2042,15 +2046,27 @@ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld( } -/* -* Clean fields data can't be used on VMLAUNCH and when we switch -* between different L2 guests as KVM keeps a single VMCS12 per L1. -*/ if (from_launch || evmcs_gpa_changed) { +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (&vmx->vcpu); +struct hv_enlightened_vmcs *evmcs = vmx->nested.hv_evmcs; + +/* +* Clean fields data can't be used on VMLAUNCH and when we +* switch between different L2 guests as KVM keeps a single +* VMCS12 per L1. +*/ vmx->nested.hv_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_ALL ; vmx->nested.force_msr_bitmap_recalc = true; + +/* +* Assume vm_id/vp_id/partition_assist_page don't change while +* eVMCS is active. +*/ + hv_vcpu->nested.pa_page_gpa = evmcs->partition_assist_page; + hv_vcpu->nested.vm_id = evmcs->hv_vm_id; + hv_vcpu->nested.vp_id = evmcs->hv_vp_id; }return EVMPTRLD_SUCCEEDED ;
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 10/19 KVM: nSVM: Keep track of Hyper-V hv_vm_id/hv_vp_id
In reply to Vitaly KuznetsovSimilar to nSVM, KVM needs to know L2's VM_ID/VP_ID and Partition assist page address to handle Direct TLB flush requests.
--- arch/x86/kvm/svm/hyperv.h | 15 +++++++++++++++ arch/x86/kvm/svm/nested.c | 2 ++ 2 files changed, 17 insertions(+)
arch/x86/kvm/svm/hyperv.h +15 -0
@@ -32,4 +32,19 @@ struct hv_enlightenments {*/ #define VMCB_HV_NESTED_ENLIGHTENMENTS VMCB_SW +static inline void nested_svm_hv_update_vm_vp_ids (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); +struct hv_enlightenments *hve = + (struct hv_enlightenments *)svm->nested.ctl.reserved_sw; +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); + +if (!hv_vcpu) +return ; + + hv_vcpu->nested.pa_page_gpa = hve->partition_assist_page; + hv_vcpu->nested.vm_id = hve->hv_vm_id; + hv_vcpu->nested.vp_id = hve->hv_vp_id; +} +#endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */
arch/x86/kvm/svm/nested.c +2 -0
@@ -752,6 +752,8 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) svm->nested.nested_run_pending =1 ; +nested_svm_hv_update_vm_vp_ids (vcpu); +if (enter_svm_guest_mode (vcpu, vmcb12_gpa, vmcb12, true)) goto out_exit_err;
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 11/19 KVM: x86: Introduce .post_hv_direct_flush() nested hook
In reply to Vitaly KuznetsovHyper-V supports injecting synthetic L2->L1 exit after performing Direct TLB flush operation but the procedure is vendor specific. Introduce .post_hv_direct_flush() nested hook for it.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/Makefile | 3 ++- arch/x86/kvm/svm/hyperv.c | 11 +++++++++++ arch/x86/kvm/svm/hyperv.h | 2 ++ arch/x86/kvm/svm/nested.c | 1 + arch/x86/kvm/vmx/evmcs.c | 4 ++++ arch/x86/kvm/vmx/evmcs.h | 1 + arch/x86/kvm/vmx/nested.c | 1 + 8 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 arch/x86/kvm/svm/hyperv.c
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1538,6 +1538,7 @@ struct kvm_x86_nested_ops {int (*enable_evmcs)(struct kvm_vcpu *vcpu,uint16_t *vmcs_version);uint16_t (*get_evmcs_version)(struct kvm_vcpu *vcpu); +void (*post_hv_direct_flush)(struct kvm_vcpu *vcpu); };struct kvm_x86_init_ops {
arch/x86/kvm/Makefile +2 -1
@@ -25,7 +25,8 @@ kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o \ vmx/evmcs.o vmx/nested.o vmx/posted_intr.o kvm-intel-$(CONFIG_X86_SGX_KVM) += vmx/sgx.o -kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o +kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o \ + svm/sev.o svm/hyperv.o ifdef CONFIG_HYPERV kvm-amd-y += svm/svm_onhyperv.o
arch/x86/kvm/svm/hyperv.c new file +11 -0
@@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* +* AMD SVM specific code for Hyper-V on KVM. +* +* Copyright 2022 Red Hat, Inc. and/or its affiliates. +*/ +#include "hyperv.h" + +void svm_post_hv_direct_flush (struct kvm_vcpu *vcpu) +{ +}
arch/x86/kvm/svm/hyperv.h +2 -0
@@ -47,4 +47,6 @@ static inline void nested_svm_hv_update_vm_vp_ids(struct kvm_vcpu *vcpu) hv_vcpu->nested.vp_id = hve->hv_vp_id; } +void svm_post_hv_direct_flush (struct kvm_vcpu *vcpu); +#endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */
arch/x86/kvm/svm/nested.c +1 -0
@@ -1573,4 +1573,5 @@ struct kvm_x86_nested_ops svm_nested_ops = { .get_nested_state_pages = svm_get_nested_state_pages, .get_state = svm_get_nested_state, .set_state = svm_set_nested_state, + .post_hv_direct_flush = svm_post_hv_direct_flush, };
arch/x86/kvm/vmx/evmcs.c +4 -0
@@ -437,3 +437,7 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,return 0 ; } + +void vmx_post_hv_direct_flush (struct kvm_vcpu *vcpu) +{ +}
arch/x86/kvm/vmx/evmcs.h +1 -0
@@ -244,5 +244,6 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version);void nested_evmcs_filter_control_msr (u32 msr_index,u64 *pdata);int nested_evmcs_check_controls (struct vmcs12 *vmcs12); +void vmx_post_hv_direct_flush (struct kvm_vcpu *vcpu);#endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/nested.c +1 -0
@@ -6828,4 +6828,5 @@ struct kvm_x86_nested_ops vmx_nested_ops = { .write_log_dirty = nested_vmx_write_pml_buffer, .enable_evmcs = nested_enable_evmcs, .get_evmcs_version = nested_get_evmcs_version, + .post_hv_direct_flush = vmx_post_hv_direct_flush, };
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 12/19 KVM: x86: hyper-v: Introduce kvm_hv_is_tlb_flush_hcall()
In reply to Vitaly KuznetsovThe newly introduced helper checks whether vCPU is performing a Hyper-V TLB flush hypercall. This is required to filter out Direct TLB flush hypercalls from L2 for processing.
--- arch/x86/kvm/hyperv.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/kvm/hyperv.h +24 -0
@@ -168,6 +168,30 @@ static inline void kvm_hv_vcpu_empty_flush_tlb(struct kvm_vcpu *vcpu) tlb_flush_ring =kvm_hv_get_tlb_flush_ring (vcpu); tlb_flush_ring->read_idx = tlb_flush_ring->write_idx; } + +static inline bool kvm_hv_is_tlb_flush_hcall (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +u16 code; + +if (!hv_vcpu) +return false; + +#ifdef CONFIG_X86_64 +if (is_64_bit_hypercall (vcpu)) { + code =kvm_rcx_read (vcpu) &0xffff ; + } else +#endif + { + code =kvm_rax_read (vcpu) &0xffff ; + } + +return (code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE || + code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST || + code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX || + code ==HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX ); +} +void kvm_hv_vcpu_flush_tlb (struct kvm_vcpu *vcpu);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 13/19 KVM: x86: hyper-v: Direct TLB flush
In reply to Vitaly KuznetsovHandle Direct TLB flush requests from L2 by going through all vCPUs and checking whether there are vCPUs running the same VM_ID with a VP_ID specified in the requests. Perform synthetic exit to L2 upon finish.
Note, while checking VM_ID/VP_ID of running vCPUs seem to be a bit racy, we count on the fact that KVM flushes the whole L2 VPID upon transition. Also, KVM_REQ_HV_TLB_FLUSH request needs to be done upon transition between L1 and L2 to make sure all pending requests are always processed.
Note, while nVMX/nSVM code does not handle VMCALL/VMMCALL from L2 yet.
--- arch/x86/kvm/hyperv.c | 65 ++++++++++++++++++++++++++++++++++++------- arch/x86/kvm/trace.h | 21 ++++++++------ 2 files changed, 68 insertions(+), 18 deletions(-)
arch/x86/kvm/hyperv.c +55 -10
@@ -34,6 +34,7 @@#include <linux/eventfd.h> #include <asm/apicdef.h> +#include <asm/mshyperv.h> #include <trace/events/kvm.h> #include "trace.h"
@@ -1863,8 +1864,8 @@ static inline int hv_tlb_flush_ring_free(struct kvm_vcpu_hv *hv_vcpu,return read_idx - write_idx -1 ; } -static void hv_tlb_flush_ring_enqueue (struct kvm_vcpu *vcpu,bool flush_all, -u64 *entries,int count) +static void hv_tlb_flush_ring_enqueue (struct kvm_vcpu *vcpu,bool direct, +bool flush_all,u64 *entries,int count) {struct kvm_vcpu_hv_tlbflush_ring *tlb_flush_ring;struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);
@@ -1875,7 +1876,7 @@ static void hv_tlb_flush_ring_enqueue(struct kvm_vcpu *vcpu, bool flush_all,if (!hv_vcpu)return ; - tlb_flush_ring = &hv_vcpu->tlb_flush_ring[0 ]; + tlb_flush_ring = direct ? &hv_vcpu->tlb_flush_ring[1 ] : &hv_vcpu->tlb_flush_ring[0 ];spin_lock_irqsave (&tlb_flush_ring->write_lock, flags);
@@ -2010,7 +2011,8 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) }trace_kvm_hv_flush_tlb (flush.processor_mask, - flush.address_space, flush.flags); + flush.address_space, flush.flags, +is_guest_mode (vcpu)); valid_bank_mask =BIT_ULL (0 ); sparse_banks[0 ] = flush.processor_mask;
@@ -2041,7 +2043,7 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)trace_kvm_hv_flush_tlb_ex (flush_ex.hv_vp_set.valid_bank_mask, flush_ex.hv_vp_set.format, flush_ex.address_space, - flush_ex.flags); + flush_ex.flags,is_guest_mode (vcpu)); valid_bank_mask = flush_ex.hv_vp_set.valid_bank_mask; all_cpus = flush_ex.hv_vp_set.format !=
@@ -2080,19 +2082,45 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)* vcpu->arch.cr3 may not be up-to-date for running vCPUs so we can't * analyze it here, flush TLB regardless of the specified address space. */ -if (all_cpus) { +if (all_cpus && !is_guest_mode (vcpu)) {kvm_for_each_vcpu (i, v, kvm) -hv_tlb_flush_ring_enqueue (v, all_addr, entries, hc->rep_cnt); +hv_tlb_flush_ring_enqueue (v, false, all_addr, entries, hc->rep_cnt);kvm_make_all_cpus_request (kvm,KVM_REQ_HV_TLB_FLUSH ); - }else { + }else if (!is_guest_mode (vcpu)) {sparse_set_to_vcpu_mask (kvm, sparse_banks, valid_bank_mask, vcpu_mask);for_each_set_bit (i, vcpu_mask,KVM_MAX_VCPUS ) { v =kvm_get_vcpu (kvm, i);if (!v)continue ; -hv_tlb_flush_ring_enqueue (v, all_addr, entries, hc->rep_cnt); +hv_tlb_flush_ring_enqueue (v, false, all_addr, entries, hc->rep_cnt); + } + +kvm_make_vcpus_request_mask (kvm,KVM_REQ_HV_TLB_FLUSH , vcpu_mask); + }else { +struct kvm_vcpu_hv *hv_v; + +bitmap_zero (vcpu_mask,KVM_MAX_VCPUS ); + +kvm_for_each_vcpu (i, v, kvm) { + hv_v =to_hv_vcpu (v); + +/* +* TLB is fully flushed on L2 VM change: either by KVM +* (on a eVMPTR switch) or by L1 hypervisor (in case it +* re-purposes the active eVMCS for a different VM/VP). +*/ +if (!hv_v || hv_v->nested.vm_id != hv_vcpu->nested.vm_id) +continue ; + +if (!all_cpus && + !hv_is_vp_in_sparse_set (hv_v->nested.vp_id, valid_bank_mask, + sparse_banks)) +continue ; + +__set_bit (i, vcpu_mask); +hv_tlb_flush_ring_enqueue (v, true, all_addr, entries, hc->rep_cnt); }kvm_make_vcpus_request_mask (kvm,KVM_REQ_HV_TLB_FLUSH , vcpu_mask);
@@ -2280,10 +2308,27 @@ static void kvm_hv_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result)static int kvm_hv_hypercall_complete (struct kvm_vcpu *vcpu,u64 result) { +int ret; +trace_kvm_hv_hypercall_done (result);kvm_hv_hypercall_set_result (vcpu, result); ++vcpu->stat.hypercalls; -return kvm_skip_emulated_instruction (vcpu); + ret =kvm_skip_emulated_instruction (vcpu); + +if (unlikely (hv_result_success (result) &&is_guest_mode (vcpu) + &&kvm_hv_is_tlb_flush_hcall (vcpu))) { +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); +u32 tlb_lock_count; + +if (unlikely (kvm_read_guest (vcpu->kvm, hv_vcpu->nested.pa_page_gpa, + &tlb_lock_count,sizeof (tlb_lock_count)))) +kvm_inject_gp (vcpu,0 ); + +if (tlb_lock_count) + kvm_x86_ops.nested_ops->post_hv_direct_flush (vcpu); + } + +return ret; }static int kvm_hv_hypercall_complete_userspace (struct kvm_vcpu *vcpu)
arch/x86/kvm/trace.h +13 -8
@@ -1477,38 +1477,41 @@ TRACE_EVENT(kvm_hv_timer_state,* Tracepoint for kvm_hv_flush_tlb. */ TRACE_EVENT (kvm_hv_flush_tlb, - TP_PROTO(u64 processor_mask, u64 address_space, u64 flags), -TP_ARGS (processor_mask, address_space, flags), + TP_PROTO(u64 processor_mask, u64 address_space, u64 flags, bool direct), +TP_ARGS (processor_mask, address_space, flags, direct),TP_STRUCT__entry ( __field(u64, processor_mask) __field(u64, address_space) __field(u64, flags) +__field (bool, direct) ),TP_fast_assign ( __entry->processor_mask = processor_mask; __entry->address_space = address_space; __entry->flags = flags; + __entry->direct = direct; ), -TP_printk ("processor_mask 0x%llx address_space 0x%llx flags 0x%llx" , +TP_printk ("processor_mask 0x%llx address_space 0x%llx flags 0x%llx %s" , __entry->processor_mask, __entry->address_space, - __entry->flags) + __entry->flags, __entry->direct ?"(direct)" :"" ) );/* * Tracepoint for kvm_hv_flush_tlb_ex. */ TRACE_EVENT(kvm_hv_flush_tlb_ex, - TP_PROTO(u64 valid_bank_mask, u64 format, u64 address_space, u64 flags), - TP_ARGS(valid_bank_mask, format, address_space, flags), + TP_PROTO(u64 valid_bank_mask, u64 format, u64 address_space, u64 flags, bool direct), + TP_ARGS(valid_bank_mask, format, address_space, flags, direct), TP_STRUCT__entry( __field(u64, valid_bank_mask) __field(u64, format) __field(u64, address_space) __field(u64, flags) + __field(bool, direct) ), TP_fast_assign(
@@ -1516,12 +1519,14 @@ TRACE_EVENT(kvm_hv_flush_tlb_ex, __entry->format = format; __entry->address_space = address_space; __entry->flags = flags; + __entry->direct = direct; ),TP_printk ("valid_bank_mask 0x%llx format 0x%llx " -"address_space 0x%llx flags 0x%llx" , +"address_space 0x%llx flags 0x%llx %s" , __entry->valid_bank_mask, __entry->format, - __entry->address_space, __entry->flags) + __entry->address_space, __entry->flags, + __entry->direct ?"(direct)" :"" ) );/*
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 14/19 KVM: x86: hyper-v: Introduce fast kvm_hv_direct_tlb_flush_exposed() check
In reply to Vitaly KuznetsovIntroduce a helper to quickly check if KVM needs to handle VMCALL/VMMCALL from L2 in L0 to process Direct TLB flush requests.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/hyperv.c | 6 ++++++ arch/x86/kvm/hyperv.h | 7 +++++++ 3 files changed, 14 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -611,6 +611,7 @@ struct kvm_vcpu_hv {u32 enlightenments_eax;/* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */ u32 enlightenments_ebx;/* HYPERV_CPUID_ENLIGHTMENT_INFO.EBX */ u32 syndbg_cap_eax;/* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ +u32 nested_features_eax;/* HYPERV_CPUID_NESTED_FEATURES.EAX */ } cpuid_cache;/* Two rings for regular Hyper-V TLB flush and Direct TLB flush */
arch/x86/kvm/hyperv.c +6 -0
@@ -2270,6 +2270,12 @@ void kvm_hv_set_cpuid(struct kvm_vcpu *vcpu) hv_vcpu->cpuid_cache.syndbg_cap_eax = entry->eax; else hv_vcpu->cpuid_cache.syndbg_cap_eax = 0; + + entry =kvm_find_cpuid_entry (vcpu,HYPERV_CPUID_NESTED_FEATURES ,0 ); +if (entry) + hv_vcpu->cpuid_cache.nested_features_eax = entry->eax; +else + hv_vcpu->cpuid_cache.nested_features_eax =0 ; }int kvm_hv_set_enforce_cpuid (struct kvm_vcpu *vcpu,bool enforce)
arch/x86/kvm/hyperv.h +7 -0
@@ -169,6 +169,13 @@ static inline void kvm_hv_vcpu_empty_flush_tlb(struct kvm_vcpu *vcpu) tlb_flush_ring->read_idx = tlb_flush_ring->write_idx; } +static inline bool kvm_hv_direct_tlb_flush_exposed (struct kvm_vcpu *vcpu) +{ +struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu); + +return hv_vcpu && (hv_vcpu->cpuid_cache.nested_features_eax &HV_X64_NESTED_DIRECT_FLUSH ); +} +static inline bool kvm_hv_is_tlb_flush_hcall (struct kvm_vcpu *vcpu) {struct kvm_vcpu_hv *hv_vcpu =to_hv_vcpu (vcpu);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 15/19 x86/hyperv: Fix 'struct hv_enlightened_vmcs' definition
In reply to Vitaly KuznetsovSection 1.9 of TLFS v6.0b says:
"All structures are padded in such a way that fields are aligned naturally (that is, an 8-byte field is aligned to an offset of 8 bytes and so on)".
'struct enlightened_vmcs' has a glitch:
...
struct {
u32 nested_flush_hypercall:1; /* 836: 0 4 */
u32 msr_bitmap:1; /* 836: 1 4 */
u32 reserved:30; /* 836: 2 4 */
} hv_enlightenments_control; /* 836 4 */
u32 hv_vp_id; /* 840 4 */
u64 hv_vm_id; /* 844 8 */
u64 partition_assist_page; /* 852 8 */...
And the observed values in 'partition_assist_page' make no sense at all. Fix the layout by padding the structure properly.
--- arch/x86/include/asm/hyperv-tlfs.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/hyperv-tlfs.h +2 -2
@@ -548,7 +548,7 @@ struct hv_enlightened_vmcs {u64 guest_rip;u32 hv_clean_fields; -u32 hv_padding_32; +u32 padding32_1;u32 hv_synthetic_controls;struct {u32 nested_flush_hypercall:1 ;
@@ -556,7 +556,7 @@ struct hv_enlightened_vmcs { u32 reserved:30; }__packed hv_enlightenments_control;u32 hv_vp_id; - +u32 padding32_2;u64 hv_vm_id;u64 partition_assist_page;u64 padding64_4[4 ];
Signature · 1 line
2.35.1
RE: [PATCH RFC 15/19] x86/hyperv: Fix 'struct hv_enlightened_vmcs' definition
From: Vitaly Kuznetsov <[email protected]> Sent: Monday, March 7, 2022 6:50 AM
Quoted 53 lines
Section 1.9 of TLFS v6.0b says:
"All structures are padded in such a way that fields are aligned naturally (that is, an 8-byte field is aligned to an offset of 8 bytes and so on)".
'struct
enlightened_vmcs' has a glitch:...
struct { u32 nested_flush_hypercall:1; /* 836: 0 4 */ u32 msr_bitmap:1; /* 836: 1 4 */ u32 reserved:30; /* 836: 2 4 */ } hv_enlightenments_control; /* 836 4 */ u32 hv_vp_id; /* 840 4 */ u64 hv_vm_id; /* 844 8 */ u64 partition_assist_page; /* 852 8 */...
And the observed values in '
partition_assist_page' make no sense at all. Fix the layout by padding the structure properly.Fixes 68d1eb72ee99 ("x86/hyper-v: define struct hv_enlightened_vmcs and cleanfield bits")
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/include/asm/hyperv-tlfs.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)arch/x86/include/asm/hyperv-tlfs.h +2
-3@@ -548,7 +548,7 @@ struct hv_enlightened_vmcs {u64 guest_rip;u32 hv_clean_fields; -u32 hv_padding_32; +u32 padding32_1;u32 hv_synthetic_controls;struct {u32 nested_flush_hypercall:1 ;@@ -556,7 +556,7 @@ struct hv_enlightened_vmcs { u32 reserved:30; }__packed hv_enlightenments_control;u32 hv_vp_id; - +u32 padding32_2;u64 hv_vm_id;u64 partition_assist_page;u64 padding64_4[4 ]; --
Quoted 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 16/19 KVM: nVMX: hyper-v: Direct TLB flush
In reply to Vitaly KuznetsovEnable Direct TLB flush feature on nVMX when:
- Enlightened VMCS is in use.
- Direct TLB flush flag is enabled in eVMCS.
- Direct TLB flush is enabled in partition assist page.
Perform synthetic vmexit to L1 after processing TLB flush call upon request (HV_VMX_SYNTHETIC_EXIT_REASON_TRAP_AFTER_FLUSH).
--- arch/x86/kvm/vmx/evmcs.c | 20 ++++++++++++++++++++ arch/x86/kvm/vmx/evmcs.h | 3 +++ arch/x86/kvm/vmx/nested.c | 16 ++++++++++++++++ 3 files changed, 39 insertions(+)
arch/x86/kvm/vmx/evmcs.c +20 -0
@@ -6,6 +6,7 @@#include "../hyperv.h" #include "../cpuid.h" #include "evmcs.h" +#include "nested.h" #include "vmcs.h" #include "vmx.h" #include "trace.h"
@@ -438,6 +439,25 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu,return 0 ; } +bool nested_evmcs_direct_flush_enabled (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); +struct hv_enlightened_vmcs *evmcs = vmx->nested.hv_evmcs; +struct hv_vp_assist_page assist_page; + +if (!evmcs) +return false; + +if (!evmcs->hv_enlightenments_control.nested_flush_hypercall) +return false; + +if (unlikely (!kvm_hv_get_assist_page (vcpu, &assist_page))) +return false; + +return assist_page.nested_control.features.directhypercall; +} +void vmx_post_hv_direct_flush (struct kvm_vcpu *vcpu) { +nested_vmx_vmexit (vcpu,HV_VMX_SYNTHETIC_EXIT_REASON_TRAP_AFTER_FLUSH ,0 ,0 ); }
arch/x86/kvm/vmx/evmcs.h +3 -0
@@ -65,6 +65,8 @@ DECLARE_STATIC_KEY_FALSE(enable_evmcs);#define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL)#define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING) +#define HV_VMX_SYNTHETIC_EXIT_REASON_TRAP_AFTER_FLUSH 0x10000031 +struct evmcs_field {u16 offset;u16 clean_field;
@@ -244,6 +246,7 @@ int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version);void nested_evmcs_filter_control_msr (u32 msr_index,u64 *pdata);int nested_evmcs_check_controls (struct vmcs12 *vmcs12); +bool nested_evmcs_direct_flush_enabled (struct kvm_vcpu *vcpu);void vmx_post_hv_direct_flush (struct kvm_vcpu *vcpu);#endif /* __KVM_X86_VMX_EVMCS_H */
arch/x86/kvm/vmx/nested.c +16 -0
@@ -1168,6 +1168,17 @@ static void nested_vmx_transition_tlb_flush(struct kvm_vcpu *vcpu, {struct vcpu_vmx *vmx =to_vmx (vcpu); +/* +* KVM_REQ_HV_TLB_FLUSH flushes entries from either L1's VPID or +* L2's VPID upon request from the guest. Make sure we check for +* pending entries for the case when the request got misplaced (e.g. +* a transition from L2->L1 happened while processing Direct TLB flush +* request or vice versa). kvm_hv_vcpu_flush_tlb() will not flush +* anything if there are no requests in the corresponding buffer. +*/ +if (to_hv_vcpu (vcpu)) +kvm_make_request (KVM_REQ_HV_TLB_FLUSH , vcpu); +/* * If vmcs12 doesn't use VPID, L1 expects linear and combined mappings * for *all* contexts to be flushed on VM-Enter/VM-Exit, i.e. it's a
@@ -5976,6 +5987,11 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu,* Handle L2's bus locks in L0 directly. */ return true; +case EXIT_REASON_VMCALL : +/* Hyper-V Direct TLB flush hypercall is handled by L0 */ +return kvm_hv_direct_tlb_flush_exposed (vcpu) && +nested_evmcs_direct_flush_enabled (vcpu) && +kvm_hv_is_tlb_flush_hcall (vcpu);default :break ; }
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 17/19 KVM: x86: KVM_REQ_TLB_FLUSH_CURRENT is a superset of KVM_REQ_HV_TLB_FLUSH too
In reply to Vitaly KuznetsovKVM_REQ_TLB_FLUSH_CURRENT is an even stronger operation than KVM_REQ_TLB_FLUSH_GUEST so KVM_REQ_HV_TLB_FLUSH needs not to be processed after it.
--- arch/x86/kvm/x86.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kvm/x86.c +4 -1
@@ -3336,8 +3336,11 @@ static inline void kvm_vcpu_flush_tlb_current(struct kvm_vcpu *vcpu)*/ void kvm_service_local_tlb_flush_requests (struct kvm_vcpu *vcpu) { -if (kvm_check_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu)) +if (kvm_check_request (KVM_REQ_TLB_FLUSH_CURRENT , vcpu)) {kvm_vcpu_flush_tlb_current (vcpu); +if (kvm_check_request (KVM_REQ_HV_TLB_FLUSH , vcpu)) +kvm_hv_vcpu_empty_flush_tlb (vcpu); + }if (kvm_check_request (KVM_REQ_TLB_FLUSH_GUEST , vcpu)) {kvm_vcpu_flush_tlb_guest (vcpu);
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 18/19 KVM: nSVM: hyper-v: Direct TLB flush
In reply to Vitaly KuznetsovImplement Hyper-V Direct TLB flush for nSVM feature. The feature needs to be enabled both in extended 'nested controls' in VMCB and partition assist page. According to TLFS, synthetic vmexit to L1 is performed with
- HV_SVM_EXITCODE_ENL exit_code.
- HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH exit_info_1.
--- arch/x86/kvm/svm/hyperv.c | 7 +++++++ arch/x86/kvm/svm/hyperv.h | 20 ++++++++++++++++++++ arch/x86/kvm/svm/nested.c | 22 +++++++++++++++++++++- 3 files changed, 48 insertions(+), 1 deletion(-)
arch/x86/kvm/svm/hyperv.c +7 -0
@@ -8,4 +8,11 @@void svm_post_hv_direct_flush (struct kvm_vcpu *vcpu) { +struct vcpu_svm *svm =to_svm (vcpu); + + svm->vmcb->control.exit_code =HV_SVM_EXITCODE_ENL ; + svm->vmcb->control.exit_code_hi =0 ; + svm->vmcb->control.exit_info_1 =HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH ; + svm->vmcb->control.exit_info_2 =0 ; +nested_svm_vmexit (svm); }
arch/x86/kvm/svm/hyperv.h +20 -0
@@ -9,6 +9,7 @@#include <asm/mshyperv.h> #include "../hyperv.h" +#include "svm.h" /* * Hyper-V uses the software reserved 32 bytes in VMCB
@@ -32,6 +33,9 @@ struct hv_enlightenments {*/ #define VMCB_HV_NESTED_ENLIGHTENMENTS VMCB_SW +#define HV_SVM_EXITCODE_ENL 0xF0000000 +#define HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH (1) +static inline void nested_svm_hv_update_vm_vp_ids (struct kvm_vcpu *vcpu) {struct vcpu_svm *svm =to_svm (vcpu);
@@ -47,6 +51,22 @@ static inline void nested_svm_hv_update_vm_vp_ids(struct kvm_vcpu *vcpu) hv_vcpu->nested.vp_id = hve->hv_vp_id; } +static inline bool nested_svm_direct_flush_enabled (struct kvm_vcpu *vcpu) +{ +struct vcpu_svm *svm =to_svm (vcpu); +struct hv_enlightenments *hve = + (struct hv_enlightenments *)svm->nested.ctl.reserved_sw; +struct hv_vp_assist_page assist_page; + +if (unlikely (!kvm_hv_get_assist_page (vcpu, &assist_page))) +return false; + +if (!hve->hv_enlightenments_control.nested_flush_hypercall) +return false; + +return assist_page.nested_control.features.directhypercall; +} +void svm_post_hv_direct_flush (struct kvm_vcpu *vcpu);#endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */
arch/x86/kvm/svm/nested.c +21 -1
@@ -153,7 +153,8 @@ void recalc_intercepts(struct vcpu_svm *svm) }/* We don't want to see VMMCALLs from a nested guest */ -vmcb_clr_intercept (c,INTERCEPT_VMMCALL ); +if (!nested_svm_direct_flush_enabled (&svm->vcpu)) +vmcb_clr_intercept (c,INTERCEPT_VMMCALL );for (i =0 ; i <MAX_INTERCEPT ; i++) c->intercepts[i] |= g->intercepts[i];
@@ -461,6 +462,17 @@ static inline bool nested_npt_enabled(struct vcpu_svm *svm)static void nested_svm_transition_tlb_flush (struct kvm_vcpu *vcpu) { +/* +* KVM_REQ_HV_TLB_FLUSH flushes entries from either L1's VPID or +* L2's VPID upon request from the guest. Make sure we check for +* pending entries for the case when the request got misplaced (e.g. +* a transition from L2->L1 happened while processing Direct TLB flush +* request or vice versa). kvm_hv_vcpu_flush_tlb() will not flush +* anything if there are no requests in the corresponding buffer. +*/ +if (to_hv_vcpu (vcpu)) +kvm_make_request (KVM_REQ_HV_TLB_FLUSH , vcpu); +/* * TODO: optimize unconditional TLB flush/MMU sync. A partial list of * things to fix before this can be conditional:
@@ -1266,6 +1278,7 @@ static int svm_check_nested_events(struct kvm_vcpu *vcpu)int nested_svm_exit_special (struct vcpu_svm *svm) {u32 exit_code = svm->vmcb->control.exit_code; +struct kvm_vcpu *vcpu = &svm->vcpu;switch (exit_code) {case SVM_EXIT_INTR :
@@ -1284,6 +1297,13 @@ int nested_svm_exit_special(struct vcpu_svm *svm)return NESTED_EXIT_HOST ;break ; } +case SVM_EXIT_VMMCALL : +/* Hyper-V Direct TLB flush hypercall is handled by L0 */ +if (kvm_hv_direct_tlb_flush_exposed (vcpu) && +nested_svm_direct_flush_enabled (vcpu) && +kvm_hv_is_tlb_flush_hcall (vcpu)) +return NESTED_EXIT_HOST ; +break ;default :break ; }
Signature · 1 line
2.35.1
Vitaly Kuznetsov
maintainerRFC PATCH 19/19 KVM: x86: Expose Hyper-V Direct TLB flush feature
In reply to Vitaly KuznetsovWith both nSVM and nVMX implementations in place, KVM can export Hyper-V Direct TLB flush feature to userspace.
--- arch/x86/kvm/hyperv.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kvm/hyperv.c +1 -0
@@ -2815,6 +2815,7 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,case HYPERV_CPUID_NESTED_FEATURES : ent->eax = evmcs_ver; + ent->eax |=HV_X64_NESTED_DIRECT_FLUSH ; ent->eax |=HV_X64_NESTED_MSR_BITMAP ;break ;
Signature · 1 line
2.35.1