i386: Enable newly introduced KVM Hyper-V enlightenments
v3 cover i386: Enable newly introduced KVM Hyper-V enlightenments
This is a continuation of "[PATCH v2 0/3] i386: Add support for Hyper-V Enlightened MSR-Bitmap and XMM fast hypercall input features":
https://lore.kernel.org/qemu-devel/[email protected]/
work which wasn't merged for 7.0, thus 'v3'.
This series enables four new KVM Hyper-V enlightenmtes:
'XMM fast hypercall input feature' is supported by KVM since v5.14, it allows for faster Hyper-V hypercall processing.
'Enlightened MSR-Bitmap' is a new nested specific enlightenment speeds up L2 vmexits by avoiding unnecessary updates to L2 MSR-Bitmap. KVM support for the feature on Intel CPUs is in v5.17 and in 5.18 for AMD CPUs.
'Extended GVA ranges for TLB flush hypercalls' indicates that extended GVA ranges are allowed to be passed to Hyper-V TLB flush hypercalls.
'Direct TLB flush hypercall' features allows L0 (KVM) to directly handle L2's TLB flush hypercalls without the need to exit to L1 (Hyper-V).
The last two features are not merged in KVM yet:
https://lore.kernel.org/kvm/[email protected]/
however, there's no direct dependency on the kernel part as thanks to KVM_GET_SUPPORTED_HV_CPUID no new capabilities are introduced.
Vitaly Kuznetsov (5):
i386: Use hv_build_cpuid_leaf() for HV_CPUID_NESTED_FEATURES
i386: Hyper-V Enlightened MSR bitmap feature
i386: Hyper-V XMM fast hypercall input feature
i386: Hyper-V Support extended GVA ranges for TLB flush hypercalls
i386: Hyper-V Direct TLB flush hypercall
docs/hyperv.txt | 34 ++++++++++++++++++++++ target/i386/cpu.c | 8 +++++ target/i386/cpu.h | 5 +++- target/i386/kvm/hyperv-proto.h | 9 +++++- target/i386/kvm/kvm.c | 53 +++++++++++++++++++++++++++++----- 5 files changed, 99 insertions(+), 10 deletions(-)
Signature · 1 line
2.35.1
PATCH v3 1/5 i386: Use hv_build_cpuid_leaf() for HV_CPUID_NESTED_FEATURES
Previously, HV_CPUID_NESTED_FEATURES.EAX CPUID leaf was handled differently as it was only used to encode the supported eVMCS version range. In fact, there are also feature (e.g. Enlightened MSR-Bitmap) bits there. In preparation to adding these features, move HV_CPUID_NESTED_FEATURES leaf handling to hv_build_cpuid_leaf() and drop now-unneeded 'hyperv_nested'.
No functional change intended.
--- target/i386/cpu.h | 1 - target/i386/kvm/kvm.c | 23 +++++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-)
target/i386/cpu.h +0 -1
@@ -1770,7 +1770,6 @@ struct ArchCPU {uint32_t hyperv_vendor_id[3 ];uint32_t hyperv_interface_id[4 ];uint32_t hyperv_limits[3 ]; -uint32_t hyperv_nested[4 ];bool hyperv_enforce_cpuid;uint32_t hyperv_ver_id_build;uint16_t hyperv_ver_id_major;
target/i386/kvm/kvm.c +15 -8
@@ -834,6 +834,8 @@ static bool tsc_is_stable_and_known(CPUX86State *env) || env->user_tsc_khz; } +#define DEFAULT_EVMCS_VERSION ((1 << 8) | 1) + static struct { const char *desc; struct {
@@ -1241,6 +1243,13 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg) } } +/* HV_CPUID_NESTED_FEATURES.EAX also encodes the supported eVMCS range */ +if (func ==HV_CPUID_NESTED_FEATURES && reg ==R_EAX ) { +if (hyperv_feat_enabled (cpu,HYPERV_FEAT_EVMCS )) { + r |=DEFAULT_EVMCS_VERSION ; + } + } +return r; }
@@ -1370,11 +1379,13 @@ static int hyperv_fill_cpuids(CPUState *cs,X86CPU *cpu =X86_CPU (cs);struct kvm_cpuid_entry2 *c;uint32_t cpuid_i =0 ; +uint32_t nested_eax = +hv_build_cpuid_leaf (cs,HV_CPUID_NESTED_FEATURES ,R_EAX ); c = &cpuid_ent[cpuid_i++]; c->function =HV_CPUID_VENDOR_AND_MAX_FUNCTIONS ; - c->eax =hyperv_feat_enabled (cpu,HYPERV_FEAT_EVMCS ) ? -HV_CPUID_NESTED_FEATURES :HV_CPUID_IMPLEMENT_LIMITS ; + c->eax = nested_eax ?HV_CPUID_NESTED_FEATURES : +HV_CPUID_IMPLEMENT_LIMITS ; c->ebx = cpu->hyperv_vendor_id[0 ]; c->ecx = cpu->hyperv_vendor_id[1 ]; c->edx = cpu->hyperv_vendor_id[2 ];
@@ -1438,7 +1449,7 @@ static int hyperv_fill_cpuids(CPUState *cs, c->ecx = cpu->hyperv_limits[1 ]; c->edx = cpu->hyperv_limits[2 ]; -if (hyperv_feat_enabled (cpu,HYPERV_FEAT_EVMCS )) { +if (nested_eax) {uint32_t function;/* Create zeroed 0x40000006..0x40000009 leaves */
@@ -1450,7 +1461,7 @@ static int hyperv_fill_cpuids(CPUState *cs, c = &cpuid_ent[cpuid_i++]; c->function =HV_CPUID_NESTED_FEATURES ; - c->eax = cpu->hyperv_nested[0 ]; + c->eax = nested_eax; }return cpuid_i;
@@ -1472,8 +1483,6 @@ static bool evmcs_version_supported(uint16_t evmcs_version, (max_version <= max_supported_version); } -#define DEFAULT_EVMCS_VERSION ((1 << 8) | 1) -static int hyperv_init_vcpu (X86CPU *cpu) {CPUState *cs =CPU (cpu);
@@ -1577,8 +1586,6 @@ static int hyperv_init_vcpu(X86CPU *cpu) supported_evmcs_version >> 8);return -ENOTSUP ; } - - cpu->hyperv_nested[0 ] = evmcs_version; }if (cpu->hyperv_enforce_cpuid) {
Signature · 1 line
2.35.1
PATCH v3 2/5 i386: Hyper-V Enlightened MSR bitmap feature
In reply to Vitaly KuznetsovThe newly introduced enlightenment allow L0 (KVM) and L1 (Hyper-V) hypervisors to collaborate to avoid unnecessary updates to L2 MSR-Bitmap upon vmexits.
--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm/hyperv-proto.h | 5 +++++ target/i386/kvm/kvm.c | 7 +++++++ 5 files changed, 25 insertions(+)
docs/hyperv.txt +10 -0
@@ -225,6 +225,16 @@ default (WS2016). Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V identification when specified without any other enlightenments. +3.21. hv-emsr-bitmap +===================== +The enlightenment is nested specific, it targets Hyper-V on KVM guests. When +enabled, it allows L0 (KVM) and L1 (Hyper-V) hypervisors to collaborate to +avoid unnecessary updates to L2 MSR-Bitmap upon vmexits. While the protocol is +supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires +Enlightened VMCS ('hv-evmcs') feature to also be enabled. + +Recommended: hv-evmcs (Intel) + 4. Supplementary features =========================
target/i386/cpu.c +2 -0
@@ -6925,6 +6925,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_STIMER_DIRECT, 0),DEFINE_PROP_BIT64 ("hv-avic" ,X86CPU , hyperv_features,HYPERV_FEAT_AVIC ,0 ), +DEFINE_PROP_BIT64 ("hv-emsr-bitmap" ,X86CPU , hyperv_features, +HYPERV_FEAT_MSR_BITMAP ,0 ),DEFINE_PROP_ON_OFF_AUTO ("hv-no-nonarch-coresharing" ,X86CPU , hyperv_no_nonarch_cs,ON_OFF_AUTO_OFF ), DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough, false),
target/i386/cpu.h +1 -0
@@ -1084,6 +1084,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define HYPERV_FEAT_IPI 13#define HYPERV_FEAT_STIMER_DIRECT 14#define HYPERV_FEAT_AVIC 15 +#define HYPERV_FEAT_MSR_BITMAP 16#ifndef HYPERV_SPINLOCK_NEVER_NOTIFY #define HYPERV_SPINLOCK_NEVER_NOTIFY 0xFFFFFFFF
target/i386/kvm/hyperv-proto.h +5 -0
@@ -72,6 +72,11 @@#define HV_ENLIGHTENED_VMCS_RECOMMENDED (1u << 14)#define HV_NO_NONARCH_CORESHARING (1u << 18) +/* +* HV_CPUID_NESTED_FEATURES.EAX bits +*/ +#define HV_NESTED_MSR_BITMAP (1u << 19) +/* * Basic virtualized MSRs */
target/i386/kvm/kvm.c +7 -0
@@ -966,6 +966,13 @@ static struct { .bits = HV_DEPRECATING_AEOI_RECOMMENDED} } }, + [HYPERV_FEAT_MSR_BITMAP ] = { + .desc ="enlightened MSR-Bitmap (hv-emsr-bitmap)" , + .flags = { + {.func =HV_CPUID_NESTED_FEATURES , .reg =R_EAX , + .bits =HV_NESTED_MSR_BITMAP } + } + }, }; static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
Signature · 1 line
2.35.1
PATCH v3 3/5 i386: Hyper-V XMM fast hypercall input feature
In reply to Vitaly KuznetsovHyper-V specification allows to pass parameters for certain hypercalls using XMM registers ("XMM Fast Hypercall Input"). When the feature is in use, it allows for faster hypercalls processing as KVM can avoid reading guest's memory.
KVM supports the feature since v5.14.
Rename HV_HYPERCALL_{PARAMS_XMM_AVAILABLE -> XMM_INPUT_AVAILABLE} to comply with KVM.
--- docs/hyperv.txt | 6 ++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm/hyperv-proto.h | 2 +- target/i386/kvm/kvm.c | 7 +++++++ 5 files changed, 17 insertions(+), 1 deletion(-)
docs/hyperv.txt +6 -0
@@ -235,6 +235,12 @@ Enlightened VMCS ('hv-evmcs') feature to also be enabled. Recommended: hv-evmcs (Intel) +3.22. hv-xmm-input +=================== +Hyper-V specification allows to pass parameters for certain hypercalls using XMM +registers ("XMM Fast Hypercall Input"). When the feature is in use, it allows +for faster hypercalls processing as KVM can avoid reading guest's memory. + 4. Supplementary features =========================
target/i386/cpu.c +2 -0
@@ -6927,6 +6927,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_AVIC, 0),DEFINE_PROP_BIT64 ("hv-emsr-bitmap" ,X86CPU , hyperv_features,HYPERV_FEAT_MSR_BITMAP ,0 ), +DEFINE_PROP_BIT64 ("hv-xmm-input" ,X86CPU , hyperv_features, +HYPERV_FEAT_XMM_INPUT ,0 ),DEFINE_PROP_ON_OFF_AUTO ("hv-no-nonarch-coresharing" ,X86CPU , hyperv_no_nonarch_cs,ON_OFF_AUTO_OFF ), DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough, false),
target/i386/cpu.h +1 -0
@@ -1085,6 +1085,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define HYPERV_FEAT_STIMER_DIRECT 14#define HYPERV_FEAT_AVIC 15#define HYPERV_FEAT_MSR_BITMAP 16 +#define HYPERV_FEAT_XMM_INPUT 17#ifndef HYPERV_SPINLOCK_NEVER_NOTIFY #define HYPERV_SPINLOCK_NEVER_NOTIFY 0xFFFFFFFF
target/i386/kvm/hyperv-proto.h +1 -1
@@ -51,7 +51,7 @@#define HV_GUEST_DEBUGGING_AVAILABLE (1u << 1)#define HV_PERF_MONITOR_AVAILABLE (1u << 2)#define HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE (1u << 3) -#define HV_HYPERCALL_PARAMS_XMM_AVAILABLE (1u << 4) +#define HV_HYPERCALL_XMM_INPUT_AVAILABLE (1u << 4)#define HV_GUEST_IDLE_STATE_AVAILABLE (1u << 5)#define HV_FREQUENCY_MSRS_AVAILABLE (1u << 8)#define HV_GUEST_CRASH_MSR_AVAILABLE (1u << 10)
target/i386/kvm/kvm.c +7 -0
@@ -973,6 +973,13 @@ static struct { .bits = HV_NESTED_MSR_BITMAP} } }, + [HYPERV_FEAT_XMM_INPUT ] = { + .desc ="XMM fast hypercall input (hv-xmm-input)" , + .flags = { + {.func =HV_CPUID_FEATURES , .reg =R_EDX , + .bits =HV_HYPERCALL_XMM_INPUT_AVAILABLE } + } + }, }; static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
Signature · 1 line
2.35.1
PATCH v3 4/5 i386: Hyper-V Support extended GVA ranges for TLB flush hypercalls
In reply to Vitaly KuznetsovKVM kind of supported "extended GVA ranges" (up to 4095 additional GFNs per hypercall) since the implementation of Hyper-V PV TLB flush feature (Linux-4.18) as regardless of the request, full TLB flush was always performed. "Extended GVA ranges for TLB flush hypercalls" feature bit wasn't exposed then. Now, as KVM gains support for fine-grained TLB flush handling, exposing this feature starts making sense.
--- docs/hyperv.txt | 7 +++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm/hyperv-proto.h | 1 + target/i386/kvm/kvm.c | 8 ++++++++ 5 files changed, 19 insertions(+)
docs/hyperv.txt +7 -0
@@ -241,6 +241,13 @@ Hyper-V specification allows to pass parameters for certain hypercalls using XMM registers ("XMM Fast Hypercall Input"). When the feature is in use, it allows for faster hypercalls processing as KVM can avoid reading guest's memory. +3.23. hv-tlbflush-ext +===================== +Allow for extended GVA ranges to be passed to Hyper-V TLB flush hypercalls +(HvFlushVirtualAddressList/HvFlushVirtualAddressListEx). + +Requires: hv-tlbflush + 4. Supplementary features =========================
target/i386/cpu.c +2 -0
@@ -6929,6 +6929,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_MSR_BITMAP, 0),DEFINE_PROP_BIT64 ("hv-xmm-input" ,X86CPU , hyperv_features,HYPERV_FEAT_XMM_INPUT ,0 ), +DEFINE_PROP_BIT64 ("hv-tlbflush-ext" ,X86CPU , hyperv_features, +HYPERV_FEAT_TLBFLUSH_EXT ,0 ),DEFINE_PROP_ON_OFF_AUTO ("hv-no-nonarch-coresharing" ,X86CPU , hyperv_no_nonarch_cs,ON_OFF_AUTO_OFF ), DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough, false),
target/i386/cpu.h +1 -0
@@ -1086,6 +1086,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define HYPERV_FEAT_AVIC 15#define HYPERV_FEAT_MSR_BITMAP 16#define HYPERV_FEAT_XMM_INPUT 17 +#define HYPERV_FEAT_TLBFLUSH_EXT 18#ifndef HYPERV_SPINLOCK_NEVER_NOTIFY #define HYPERV_SPINLOCK_NEVER_NOTIFY 0xFFFFFFFF
target/i386/kvm/hyperv-proto.h +1 -0
@@ -55,6 +55,7 @@#define HV_GUEST_IDLE_STATE_AVAILABLE (1u << 5)#define HV_FREQUENCY_MSRS_AVAILABLE (1u << 8)#define HV_GUEST_CRASH_MSR_AVAILABLE (1u << 10) +#define HV_EXT_GVA_RANGES_FLUSH_AVAILABLE (1u << 14)#define HV_STIMER_DIRECT_MODE_AVAILABLE (1u << 19)/*
target/i386/kvm/kvm.c +8 -0
@@ -980,6 +980,14 @@ static struct { .bits = HV_HYPERCALL_XMM_INPUT_AVAILABLE} } }, + [HYPERV_FEAT_TLBFLUSH_EXT ] = { + .desc ="Extended gva ranges for TLB flush hypercalls (hv-tlbflush-ext)" , + .flags = { + {.func =HV_CPUID_FEATURES , .reg =R_EDX , + .bits =HV_EXT_GVA_RANGES_FLUSH_AVAILABLE } + }, + .dependencies =BIT (HYPERV_FEAT_TLBFLUSH ) + }, }; static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
Signature · 1 line
2.35.1
PATCH v3 5/5 i386: Hyper-V Direct TLB flush hypercall
In reply to Vitaly KuznetsovHyper-V TLFS allows for L0 and L1 hypervisors to collaborate on L2's TLB flush hypercalls handling. With the correct setup, L2's TLB flush hypercalls can be handled by L0 directly, without the need to exit to L1.
--- docs/hyperv.txt | 11 +++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm/hyperv-proto.h | 1 + target/i386/kvm/kvm.c | 8 ++++++++ 5 files changed, 23 insertions(+)
docs/hyperv.txt +11 -0
@@ -248,6 +248,17 @@ Allow for extended GVA ranges to be passed to Hyper-V TLB flush hypercalls Requires: hv-tlbflush +3.24. hv-tlbflush-direct +========================= +The enlightenment is nested specific, it targets Hyper-V on KVM guests. When +enabled, it allows L0 (KVM) to directly handle TLB flush hypercalls from L2 +guest without the need to exit to L1 (Hyper-V) hypervisor. While the feature is +supported for both VMX (Intel) and SVM (AMD), the VMX implementation requires +Enlightened VMCS ('hv-evmcs') feature to also be enabled. + +Requires: hv-vapic +Recommended: hv-evmcs (Intel) + 4. Supplementary features =========================
target/i386/cpu.c +2 -0
@@ -6931,6 +6931,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_XMM_INPUT, 0),DEFINE_PROP_BIT64 ("hv-tlbflush-ext" ,X86CPU , hyperv_features,HYPERV_FEAT_TLBFLUSH_EXT ,0 ), +DEFINE_PROP_BIT64 ("hv-tlbflush-direct" ,X86CPU , hyperv_features, +HYPERV_FEAT_TLBFLUSH_DIRECT ,0 ),DEFINE_PROP_ON_OFF_AUTO ("hv-no-nonarch-coresharing" ,X86CPU , hyperv_no_nonarch_cs,ON_OFF_AUTO_OFF ), DEFINE_PROP_BOOL("hv-passthrough", X86CPU, hyperv_passthrough, false),
target/i386/cpu.h +1 -0
@@ -1087,6 +1087,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define HYPERV_FEAT_MSR_BITMAP 16#define HYPERV_FEAT_XMM_INPUT 17#define HYPERV_FEAT_TLBFLUSH_EXT 18 +#define HYPERV_FEAT_TLBFLUSH_DIRECT 19#ifndef HYPERV_SPINLOCK_NEVER_NOTIFY #define HYPERV_SPINLOCK_NEVER_NOTIFY 0xFFFFFFFF
target/i386/kvm/hyperv-proto.h +1 -0
@@ -76,6 +76,7 @@/* * HV_CPUID_NESTED_FEATURES.EAX bits */ +#define HV_NESTED_DIRECT_FLUSH (1u << 17)#define HV_NESTED_MSR_BITMAP (1u << 19)/*
target/i386/kvm/kvm.c +8 -0
@@ -988,6 +988,14 @@ static struct { }, .dependencies = BIT(HYPERV_FEAT_TLBFLUSH) }, + [HYPERV_FEAT_TLBFLUSH_DIRECT] = { + .desc = "direct TLB flush (hv-tlbflush-direct)", + .flags = { + {.func = HV_CPUID_NESTED_FEATURES, .reg = R_EAX, + .bits = HV_NESTED_DIRECT_FLUSH} + }, + .dependencies = BIT(HYPERV_FEAT_VAPIC) + }, }; static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
Signature · 1 line
2.35.1
Quoted 3 lines
This series enables four new KVM Hyper-V enlightenmtes [...]
docs/hyperv.txt| 34 ++++++++++++++++++++++
Queued, thanks. Would you please convert hyperv.txt to rST in docs/system/i386? The various enlightenments can be converted to a definition list.
Paolo
Quoted 5 lines · Paolo Bonzini
Quoted 3 lines
This series enables four new KVM Hyper-V enlightenmtes [...]
docs/hyperv.txt| 34 ++++++++++++++++++++++Queued, thanks.
Thanks!
Quoted 1 line
Would you please convert hyperv.txt to rST in docs/system/i386?
Sure, it's on my TODO list.
Signature · 1 line
Vitaly
Quoted 10 lines · Vitaly Kuznetsov
Quoted 5 lines · Paolo Bonzini
Quoted 3 lines
This series enables four new KVM Hyper-V enlightenmtes [...]
docs/hyperv.txt| 34 ++++++++++++++++++++++Queued, thanks.
Thanks!
It seems these patches didn't make it upstream yet but there's a (small) conflict with
commit 73d24074078a2cefb5305047e3bf50b73daa3f98
Author: Jon Doron <[email protected]> Date: Wed Feb 16 12:24:59 2022 +0200
hyperv: Add support to process syndbg commands
which did.
Quoted 3 lines
Quoted 1 line
Would you please convert hyperv.txt to rST in docs/system/i386?
Sure, it's on my TODO list.
I've sent it out some time ago:
https://lore.kernel.org/qemu-devel/[email protected]/
but it also conflicts with 73d24074078a now because of 'hv-syndbg'. I'm going to send out 'v4' including the conversion to rst to (hopefully) facilitate acceptance.
Signature · 1 line
Vitaly