target/i386/kvm: Add Hyper-V direct tlb flush support
PATCH v4 1/1 target/i386/kvm: Add Hyper-V direct tlb flush support
From: Tianyu Lan <[email protected]>
Hyper-V direct tlb flush targets KVM on Hyper-V guest. Enable direct TLB flush for its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall parameters between Hyper-V and KVM, KVM capabilities should be hidden when enable Hyper-V direct tlb flush otherwise KVM hypercalls may be intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush". Check expose_kvm and Hyper-V tlb flush capability status before enabling the feature.
---
Change since v3:
- Fix logic of Hyper-V passthrough mode with direct tlb flush.
Change sicne v2:
- Update new feature description and name.
- Change failure print log.
Change since v1:
- Add direct tlb flush's Hyper-V property and use hv_cpuid_check_and_set() to check the dependency of tlbflush feature.
- Make new feature work with Hyper-V passthrough mode.
--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm.c | 24 ++++++++++++++++++++++++ 4 files changed, 37 insertions(+)
docs/hyperv.txt +10 -0
@@ -184,6 +184,16 @@ enabled. Requires: hv-vpindex, hv-synic, hv-time, hv-stimer +3.18. hv-direct-tlbflush +======================= +Enable direct TLB flush for KVM when it is running as a nested +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from L2 +guests are being passed through to L0 (Hyper-V) for handling. Due to ABI +differences between Hyper-V and KVM hypercalls, L2 guests will not be +able to issue KVM hypercalls (as those could be mishanled by L0 +Hyper-V), this requires KVM hypervisor signature to be hidden. + +Requires: hv-tlbflush, -kvm 4. Development features ========================
target/i386/cpu.c +2 -0
@@ -6156,6 +6156,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_IPI, 0),DEFINE_PROP_BIT64 ("hv-stimer-direct" ,X86CPU , hyperv_features,HYPERV_FEAT_STIMER_DIRECT ,0 ), +DEFINE_PROP_BIT64 ("hv-direct-tlbflush" ,X86CPU , hyperv_features, +HYPERV_FEAT_DIRECT_TLBFLUSH ,0 ),DEFINE_PROP_BOOL ("hv-passthrough" ,X86CPU , hyperv_passthrough, false), DEFINE_PROP_BOOL("check", X86CPU, check_cpuid, true),
target/i386/cpu.h +1 -0
@@ -907,6 +907,7 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];#define HYPERV_FEAT_EVMCS 12#define HYPERV_FEAT_IPI 13#define HYPERV_FEAT_STIMER_DIRECT 14 +#define HYPERV_FEAT_DIRECT_TLBFLUSH 15#ifndef HYPERV_SPINLOCK_NEVER_RETRY #define HYPERV_SPINLOCK_NEVER_RETRY 0xFFFFFFFF
target/i386/kvm.c +24 -0
@@ -900,6 +900,10 @@ static struct { }, .dependencies = BIT(HYPERV_FEAT_STIMER) }, + [HYPERV_FEAT_DIRECT_TLBFLUSH] = { + .desc = "direct paravirtualized TLB flush (hv-direct-tlbflush)", + .dependencies =BIT (HYPERV_FEAT_TLBFLUSH ) + }, };static struct kvm_cpuid2 *try_get_hv_cpuid (CPUState *cs,int max)
@@ -1224,6 +1228,7 @@ static int hyperv_handle_properties(CPUState *cs, r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_EVMCS ); r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_IPI ); r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_STIMER_DIRECT ); + r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_DIRECT_TLBFLUSH );/* Additional dependencies not covered by kvm_hyperv_properties[] */ if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&
@@ -1243,6 +1248,25 @@ static int hyperv_handle_properties(CPUState *cs, goto free; } +if (hyperv_feat_enabled (cpu,HYPERV_FEAT_DIRECT_TLBFLUSH )) { +if (kvm_vcpu_enable_cap (cs,KVM_CAP_HYPERV_DIRECT_TLBFLUSH ,0 ,0 )) { +if (!cpu->hyperv_passthrough) { +fprintf (stderr, +"Hyper-V %s is not supported by kernel\n" , + kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH ].desc); +return -ENOSYS ; + } + + cpu->hyperv_features &= ~BIT (HYPERV_FEAT_DIRECT_TLBFLUSH ); + }else if (cpu->expose_kvm) { +fprintf (stderr, +"Hyper-V %s requires KVM hypervisor signature " +"to be hidden (-kvm).\n" , + kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH ].desc); +return -ENOSYS ; + } + } +if (cpu->hyperv_passthrough) {/* We already copied all feature words from KVM as is */ r = cpuid->nent;
Signature · 1 line
2.14.5
Quoted 40 lines · On Tue, Nov 12, 2019 at 11:34:27AM +0800, [email protected] wrote:
From: Tianyu Lan <[email protected]>
Hyper-V direct tlb flush targets KVM on Hyper-V guest. Enable direct TLB flush for its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall parameters between Hyper-V and KVM, KVM capabilities should be hidden when enable Hyper-V direct tlb flush otherwise KVM hypercalls may be intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush". Check
expose_kvmand Hyper-V tlb flush capability status before enabling the feature.Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix logic of Hyper-V passthrough mode with direct tlb flush.Change sicne v2:
- Update new feature description and name.
- Change failure print log.Change since v1:
- Add direct tlb flush's Hyper-V property and usehv_cpuid_check_and_set()to check the dependency of tlbflush feature.
- Make new feature work with Hyper-V passthrough mode.--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm.c | 24 ++++++++++++++++++++++++ 4 files changed, 37 insertions(+)docs/hyperv.txt +0
-0@@ -184,6 +184,16 @@ enabled.
docs/hyperv.txt enabled. · line 194
Quoted 10 lines
docs/hyperv.txt +8
-0Requires: hv-vpindex, hv-synic, hv-time, hv-stimer +3.18. hv-direct-tlbflush +======================= +Enable direct TLB flush for KVM when it is running as a nested +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from L2 +guests are being passed through to L0 (Hyper-V) for handling. Due to ABI +differences between Hyper-V and KVM hypercalls, L2 guests will not be +able to issue KVM hypercalls (as those could be mishanled by L0 +Hyper-V), this requires KVM hypervisor signature to be hidden.
On a second thought, I wonder if this is the only conflict we have.
In KVM, kvm_emulate_hypercall, when sees Hyper-V hypercalls enabled, just calls kvm_hv_hypercall and returns. I.e. once the userspace enables Hyper-V hypercalls (which QEMU does when any of hv_* flags is given), KVM treats *all* hypercalls as Hyper-V ones and handles *no* KVM hypercalls.
So, if hiding the KVM hypervisor signature is the only way to prevent the guest from issuing KVM hypercalls (need to double-check), then, I'm afraid, we just need to require it as soon as any Hyper-V feature is enabled.
Quoted 63 lines
docs/hyperv.txt +1
-0+Requires: hv-tlbflush, -kvm 4. Development features ========================target/i386/cpu.c +2
-0@@ -6156,6 +6156,8 @@ static Property x86_cpu_properties[] = { HYPERV_FEAT_IPI, 0),DEFINE_PROP_BIT64 ("hv-stimer-direct" ,X86CPU , hyperv_features,HYPERV_FEAT_STIMER_DIRECT ,0 ), +DEFINE_PROP_BIT64 ("hv-direct-tlbflush" ,X86CPU , hyperv_features, +HYPERV_FEAT_DIRECT_TLBFLUSH ,0 ),DEFINE_PROP_BOOL ("hv-passthrough" ,X86CPU , hyperv_passthrough, false), DEFINE_PROP_BOOL("check", X86CPU, check_cpuid, true),target/i386/cpu.h +1
-0@@ -907,6 +907,7 @@ typedef uint64_t FeatureWordArray[FEATURE_WORDS];#define HYPERV_FEAT_EVMCS 12#define HYPERV_FEAT_IPI 13#define HYPERV_FEAT_STIMER_DIRECT 14 +#define HYPERV_FEAT_DIRECT_TLBFLUSH 15#ifndef HYPERV_SPINLOCK_NEVER_RETRY #define HYPERV_SPINLOCK_NEVER_RETRY 0xFFFFFFFFtarget/i386/kvm.c +12
-0@@ -900,6 +900,10 @@ static struct { }, .dependencies = BIT(HYPERV_FEAT_STIMER) }, + [HYPERV_FEAT_DIRECT_TLBFLUSH] = { + .desc = "direct paravirtualized TLB flush (hv-direct-tlbflush)", + .dependencies =BIT (HYPERV_FEAT_TLBFLUSH ) + }, };static struct kvm_cpuid2 *try_get_hv_cpuid (CPUState *cs,int max)@@ -1224,6 +1228,7 @@ static int hyperv_handle_properties(CPUState *cs, r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_EVMCS ); r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_IPI ); r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_STIMER_DIRECT ); + r |=hv_cpuid_check_and_set (cs, cpuid,HYPERV_FEAT_DIRECT_TLBFLUSH );/* Additional dependencies not covered by kvm_hyperv_properties[] */ if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&@@ -1243,6 +1248,25 @@ static int hyperv_handle_properties(CPUState *cs, goto free; } +if (hyperv_feat_enabled (cpu,HYPERV_FEAT_DIRECT_TLBFLUSH )) { +if (kvm_vcpu_enable_cap (cs,KVM_CAP_HYPERV_DIRECT_TLBFLUSH ,0 ,0 )) { +if (!cpu->hyperv_passthrough) { +fprintf (stderr, +"Hyper-V %s is not supported by kernel\n" , + kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH ].desc); +return -ENOSYS ;
target/i386/kvm.c static int hyperv_handle_properties(CPUState *cs, · line 1267
Quoted 10 lines
target/i386/kvm.c +10
-0+ } + + cpu->hyperv_features &= ~BIT (HYPERV_FEAT_DIRECT_TLBFLUSH ); + }else if (cpu->expose_kvm) { +fprintf (stderr, +"Hyper-V %s requires KVM hypervisor signature " +"to be hidden (-kvm).\n" , + kvm_hyperv_properties[HYPERV_FEAT_DIRECT_TLBFLUSH ].desc); +return -ENOSYS ; + }
In view of my comment above, this "else if" clause may become unnecessary.
However, it doesn't hurt either, and doesn't make things worse, so, if this is seen as 4.2 material and the general KVM vs Hyper-V hypercall conflict resolution is postponed till after 4.2, the patch looks ok as it is.
Under this provision
Quoted 8 lines
target/i386/kvm.c +2
-0+ } +if (cpu->hyperv_passthrough) {/* We already copied all feature words from KVM as is */ r = cpuid->nent;Signature · 1 line
2.14.5
Quoted 54 lines · Roman Kagan
Quoted 40 lines · On Tue, Nov 12, 2019 at 11:34:27AM +0800, [email protected] wrote:
From: Tianyu Lan <[email protected]>
Hyper-V direct tlb flush targets KVM on Hyper-V guest. Enable direct TLB flush for its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall parameters between Hyper-V and KVM, KVM capabilities should be hidden when enable Hyper-V direct tlb flush otherwise KVM hypercalls may be intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush". Check
expose_kvmand Hyper-V tlb flush capability status before enabling the feature.Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix logic of Hyper-V passthrough mode with direct tlb flush.Change sicne v2:
- Update new feature description and name.
- Change failure print log.Change since v1:
- Add direct tlb flush's Hyper-V property and usehv_cpuid_check_and_set()to check the dependency of tlbflush feature.
- Make new feature work with Hyper-V passthrough mode.--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm.c | 24 ++++++++++++++++++++++++ 4 files changed, 37 insertions(+)docs/hyperv.txt +0
-0@@ -184,6 +184,16 @@ enabled.docs/hyperv.txt enabled. · line 194
Quoted 10 lines
docs/hyperv.txt +8
-0Requires: hv-vpindex, hv-synic, hv-time, hv-stimer +3.18. hv-direct-tlbflush +======================= +Enable direct TLB flush for KVM when it is running as a nested +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from L2 +guests are being passed through to L0 (Hyper-V) for handling. Due to ABI +differences between Hyper-V and KVM hypercalls, L2 guests will not be +able to issue KVM hypercalls (as those could be mishanled by L0 +Hyper-V), this requires KVM hypervisor signature to be hidden.On a second thought, I wonder if this is the only conflict we have.
Quoted 5 lines
In KVM,
kvm_emulate_hypercall, when sees Hyper-V hypercalls enabled, just callskvm_hv_hypercalland returns. I.e. once the userspace enables Hyper-V hypercalls (which QEMU does when any of hv_* flags is given), KVM treats *all* hypercalls as Hyper-V ones and handles *no* KVM hypercalls.
Yes, but only after guest enables Hyper-V hypercalls by writing to HV_X64_MSR_HYPERCALL. E.g. if you run a Linux guest and add a couple hv_* flags on the QEMU command line the guest will still be able to use KVM hypercalls normally becase Linux won't enable Hyper-V hypercall page.
Quoted 6 lines
So, if hiding the KVM hypervisor signature is the only way to prevent the guest from issuing KVM hypercalls (need to double-check), then, I'm afraid, we just need to require it as soon as any Hyper-V feature is enabled.
If we do that we're going to break a lot of setups in the wild which run Linux guests with hv_* flags (e.g. just to keep configuration the same for Windows/Linux or by mistake/misunderstanding).
When Hyper-V enlightenments are enabled, KVM signature moves to 0x40000100 so if a guest is still able to find it -- then it knows what's going on. I'd suggest we maintain the status quo.
Signature · 1 line
Vitaly
Quoted 61 lines · Vitaly Kuznetsov
Quoted 54 lines · Roman Kagan
Quoted 40 lines · On Tue, Nov 12, 2019 at 11:34:27AM +0800, [email protected] wrote:
From: Tianyu Lan <[email protected]>
Hyper-V direct tlb flush targets KVM on Hyper-V guest. Enable direct TLB flush for its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall parameters between Hyper-V and KVM, KVM capabilities should be hidden when enable Hyper-V direct tlb flush otherwise KVM hypercalls may be intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush". Check
expose_kvmand Hyper-V tlb flush capability status before enabling the feature.Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix logic of Hyper-V passthrough mode with direct tlb flush.Change sicne v2:
- Update new feature description and name.
- Change failure print log.Change since v1:
- Add direct tlb flush's Hyper-V property and usehv_cpuid_check_and_set()to check the dependency of tlbflush feature.
- Make new feature work with Hyper-V passthrough mode.--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm.c | 24 ++++++++++++++++++++++++ 4 files changed, 37 insertions(+)docs/hyperv.txt +0
-0@@ -184,6 +184,16 @@ enabled.docs/hyperv.txt enabled. · line 194
Quoted 10 lines
docs/hyperv.txt +8
-0Requires: hv-vpindex, hv-synic, hv-time, hv-stimer +3.18. hv-direct-tlbflush +======================= +Enable direct TLB flush for KVM when it is running as a nested +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from L2 +guests are being passed through to L0 (Hyper-V) for handling. Due to ABI +differences between Hyper-V and KVM hypercalls, L2 guests will not be +able to issue KVM hypercalls (as those could be mishanled by L0 +Hyper-V), this requires KVM hypervisor signature to be hidden.On a second thought, I wonder if this is the only conflict we have.
Quoted 5 lines
In KVM,
kvm_emulate_hypercall, when sees Hyper-V hypercalls enabled, just callskvm_hv_hypercalland returns. I.e. once the userspace enables Hyper-V hypercalls (which QEMU does when any of hv_* flags is given), KVM treats *all* hypercalls as Hyper-V ones and handles *no* KVM hypercalls.
Quoted 5 lines
Yes, but only after guest enables Hyper-V hypercalls by writing to
HV_X64_MSR_HYPERCALL. E.g. if you run a Linux guest and add a couple hv_* flags on the QEMU command line the guest will still be able to use KVM hypercalls normally becase Linux won't enable Hyper-V hypercall page.
Ah, you're right. There's no conflict indeed, the guest makes deliberate choice which hypercall ABI to use.
Then QEMU (or KVM on its own?) should only activate this flag in evmcs if it sees that the guest has enabled Hyper-V hypercalls. No need to hide KVM signature.
Roman.
Quoted 68 lines · Roman Kagan
Quoted 61 lines · Vitaly Kuznetsov
Quoted 54 lines · Roman Kagan
Quoted 40 lines · On Tue, Nov 12, 2019 at 11:34:27AM +0800, [email protected] wrote:
From: Tianyu Lan <[email protected]>
Hyper-V direct tlb flush targets KVM on Hyper-V guest. Enable direct TLB flush for its guests meaning that TLB flush hypercalls are handled by Level 0 hypervisor (Hyper-V) bypassing KVM in Level 1. Due to the different ABI for hypercall parameters between Hyper-V and KVM, KVM capabilities should be hidden when enable Hyper-V direct tlb flush otherwise KVM hypercalls may be intercepted by Hyper-V. Add new parameter "hv-direct-tlbflush". Check
expose_kvmand Hyper-V tlb flush capability status before enabling the feature.Signed-off-by Tianyu Lan <[email protected]>---
Change since v3:
- Fix logic of Hyper-V passthrough mode with direct tlb flush.Change sicne v2:
- Update new feature description and name.
- Change failure print log.Change since v1:
- Add direct tlb flush's Hyper-V property and usehv_cpuid_check_and_set()to check the dependency of tlbflush feature.
- Make new feature work with Hyper-V passthrough mode.--- docs/hyperv.txt | 10 ++++++++++ target/i386/cpu.c | 2 ++ target/i386/cpu.h | 1 + target/i386/kvm.c | 24 ++++++++++++++++++++++++ 4 files changed, 37 insertions(+)docs/hyperv.txt +0
-0@@ -184,6 +184,16 @@ enabled.docs/hyperv.txt enabled. · line 194
Quoted 10 lines
docs/hyperv.txt +8
-0Requires: hv-vpindex, hv-synic, hv-time, hv-stimer +3.18. hv-direct-tlbflush +======================= +Enable direct TLB flush for KVM when it is running as a nested +hypervisor on top Hyper-V. When enabled, TLB flush hypercalls from L2 +guests are being passed through to L0 (Hyper-V) for handling. Due to ABI +differences between Hyper-V and KVM hypercalls, L2 guests will not be +able to issue KVM hypercalls (as those could be mishanled by L0 +Hyper-V), this requires KVM hypervisor signature to be hidden.On a second thought, I wonder if this is the only conflict we have.
Quoted 5 lines
In KVM,
kvm_emulate_hypercall, when sees Hyper-V hypercalls enabled, just callskvm_hv_hypercalland returns. I.e. once the userspace enables Hyper-V hypercalls (which QEMU does when any of hv_* flags is given), KVM treats *all* hypercalls as Hyper-V ones and handles *no* KVM hypercalls.Quoted 5 lines
Yes, but only after guest enables Hyper-V hypercalls by writing to
HV_X64_MSR_HYPERCALL. E.g. if you run a Linux guest and add a couple hv_* flags on the QEMU command line the guest will still be able to use KVM hypercalls normally becase Linux won't enable Hyper-V hypercall page.
Quoted 5 lines
Ah, you're right. There's no conflict indeed, the guest makes deliberate choice which hypercall ABI to use.
Then QEMU (or KVM on its own?) should only activate this flag in evmcs if it sees that the guest has enabled Hyper-V hypercalls.
That was my suggestion as well when KVM patches were submitted, but if I remember correctly Tianyu said that if we don't enable 'direct tlb flush' flag in eVMCS on first VMLAUNCH, underlying Hyper-V won't give us a second chance so we can't enadle it after guest writes to HV_X64_MSR_HYPERCALL. This is a very unfortunate design/implementation.
Signature · 1 line
Vitaly