KVM: x86: hyper-v: make L2 Hyper-V 2019 on KVM guests see MD_CLEAR
Vitaly Kuznetsov
maintainercover KVM: x86: hyper-v: make L2 Hyper-V 2019 on KVM guests see MD_CLEAR
[The series is KVM specific but the first patch of the series likely requires
someone else's ACK. hyperv-tlfs.h gets a small addition too.]
It was discovered that L2 guests on Hyper-V 2019 on KVM don't see MD_CLEAR bit (and thus think they're MDS vulnerable) even when it is present on the host. Turns out, Hyper-V is filtering it out because it is not sure the topology L0 is exposing is trustworthy and generally it is not. In some specific cases (e.g. when SMT is unsupported or forcesully disabled) it is and we can tell this to userspace hoping that it'll pass this info to L1. See PATCH2 of the series for additional details.
The series can be tested with QEMU-4.1+ and 'hv-passthrough' CPU flag.
Vitaly Kuznetsov (3):
cpu/SMT: create and export cpu_smt_possible()
KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when
SMT is impossible
KVM: selftests: hyperv_cpuid: add check for
NoNonArchitecturalCoreSharing bit
arch/x86/include/asm/hyperv-tlfs.h | 7 +++++ arch/x86/kvm/hyperv.c | 4 ++- include/linux/cpu.h | 2 ++ kernel/cpu.c | 11 ++++++-- .../selftests/kvm/x86_64/hyperv_cpuid.c | 27 +++++++++++++++++++ 5 files changed, 48 insertions(+), 3 deletions(-)
Signature · 1 line
2.20.1
Vitaly Kuznetsov
maintainerPATCH 1/3 cpu/SMT: create and export cpu_smt_possible()
KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export cpu_smt_possible() answering this question.
--- include/linux/cpu.h | 2 ++ kernel/cpu.c | 11 +++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-)
include/linux/cpu.h +2 -0
@@ -201,12 +201,14 @@ enum cpuhp_smt_control {extern enum cpuhp_smt_control cpu_smt_control;extern void cpu_smt_disable (bool force);extern void cpu_smt_check_topology (void ); +extern bool cpu_smt_possible (void );extern int cpuhp_smt_enable (void );extern int cpuhp_smt_disable (enum cpuhp_smt_control ctrlval);#else # define cpu_smt_control (CPU_SMT_NOT_IMPLEMENTED)static inline void cpu_smt_disable (bool force) { }static inline void cpu_smt_check_topology (void ) { } +static inline bool cpu_smt_possible (void ) {return false; }static inline int cpuhp_smt_enable (void ) {return 0 ; }static inline int cpuhp_smt_disable (enum cpuhp_smt_control ctrlval) {return 0 ; }#endif
kernel/cpu.c +9 -2
@@ -389,8 +389,7 @@ enum cpuhp_smt_control cpu_smt_control __read_mostly = CPU_SMT_ENABLED;void __init cpu_smt_disable (bool force) { -if (cpu_smt_control ==CPU_SMT_FORCE_DISABLED || - cpu_smt_control ==CPU_SMT_NOT_SUPPORTED ) +if (!cpu_smt_possible ())return ;if (force) {
@@ -435,6 +434,14 @@ static inline bool cpu_smt_allowed(unsigned int cpu)*/ return !per_cpu(cpuhp_state, cpu).booted_once; } + +/* Returns true if SMT is not supported of forcefully (irreversibly) disabled */ +bool cpu_smt_possible (void ) +{ +return cpu_smt_control !=CPU_SMT_FORCE_DISABLED && + cpu_smt_control !=CPU_SMT_NOT_SUPPORTED ; +} +EXPORT_SYMBOL_GPL (cpu_smt_possible);#else static inline bool cpu_smt_allowed (unsigned int cpu) {return true; }#endif
Signature · 1 line
2.20.1
Re: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()
Quoted 4 lines · Vitaly Kuznetsov
KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export
cpu_smt_possible()answering this question.
It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?
Paolo Bonzini
maintainerRe: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()
Quoted 10 lines · Jim Mattson
Quoted 3 lines
KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export
cpu_smt_possible()answering this question.It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?
Yes, however that scheduler API could also rely on something like cpu_smt_possible(), at least in the case where core scheduling is not active, so this is still a step in the right direction.
Paolo
Vitaly Kuznetsov
maintainerRe: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()
In reply to Jim MattsonQuoted 6 lines · Jim Mattson
Quoted 4 lines · Vitaly Kuznetsov
KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export
cpu_smt_possible()answering this question.
Quoted 7 lines
It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?
(I know not that much about scheduler so please bear with me)
Having a trustworthy scheduler not placing unrelated (not exposed as sibling SMT threads to a guest or vCPUs from different guests) on sibling SMT threads when it's not limited with affinity is definitely a good thing. We, however, also need to know if vCPU pinning is planned for the guest: like when QEMU vCPU threads are created they're not pinned, however, libvirt pins them if needed before launching the guest. So 'NoNonArchitecturalCoreSharing' can also be set in two cases:
- No vCPU pinning is planned but the scheduler is aware of the problem (I'm not sure it is nowadays)
- The upper layer promises to do the right pinning.
This patch series, however, doesn't go that deep, it only covers the simplest case: SMT is unavailable or forcefully disabled. I'll try to learn more about scheduler though.
Signature · 1 line
Vitaly
Vitaly Kuznetsov
maintainerPATCH 2/3 KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
In reply to Vitaly KuznetsovHyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.
NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).
This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).
--- arch/x86/include/asm/hyperv-tlfs.h | 7 +++++++ arch/x86/kvm/hyperv.c | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/hyperv-tlfs.h +7 -0
@@ -170,6 +170,13 @@/* Recommend using enlightened VMCS */ #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED BIT(14) +/* +* Virtual processor will never share a physical core with another virtual +* processor, except for virtual processors that are reported as sibling SMT +* threads. +*/ +#define HV_X64_NO_NONARCH_CORESHARING BIT(18) +/* Nested features. These are HYPERV_CPUID_NESTED_FEATURES.EAX bits. */ #define HV_X64_NESTED_GUEST_MAPPING_FLUSH BIT(18)#define HV_X64_NESTED_MSR_BITMAP BIT(19)
arch/x86/kvm/hyperv.c +3 -1
@@ -23,6 +23,7 @@#include "ioapic.h" #include "hyperv.h" +#include <linux/cpu.h> #include <linux/kvm_host.h> #include <linux/highmem.h> #include <linux/sched/cputime.h>
@@ -1864,7 +1865,8 @@ int kvm_vcpu_ioctl_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, ent->eax |=HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED ;if (evmcs_ver) ent->eax |=HV_X64_ENLIGHTENED_VMCS_RECOMMENDED ; - +if (!cpu_smt_possible ()) + ent->eax |=HV_X64_NO_NONARCH_CORESHARING ;/* * Default number of spinlock retry attempts, matches * HyperV 2016.
Signature · 1 line
2.20.1
Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
Quoted 11 lines · Vitaly Kuznetsov
Hyper-V 2019 doesn't expose
MD_CLEARCPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.
Quoted 9 lines
NoNonArchitecturalCoreSharing (CPUID
0x40000004EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).
That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?
Quoted 54 lines
This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/include/asm/hyperv-tlfs.h | 7 +++++++ arch/x86/kvm/hyperv.c | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-)arch/x86/include/asm/hyperv-tlfs.h +7
-0@@ -170,6 +170,13 @@/* Recommend using enlightened VMCS */ #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED BIT(14) +/* +* Virtual processor will never share a physical core with another virtual +* processor, except for virtual processors that are reported as sibling SMT +* threads. +*/ +#define HV_X64_NO_NONARCH_CORESHARING BIT(18) +/* Nested features. These are HYPERV_CPUID_NESTED_FEATURES.EAX bits. */ #define HV_X64_NESTED_GUEST_MAPPING_FLUSH BIT(18)#define HV_X64_NESTED_MSR_BITMAP BIT(19)arch/x86/kvm/hyperv.c +3
-2@@ -23,6 +23,7 @@#include "ioapic.h" #include "hyperv.h" +#include <linux/cpu.h> #include <linux/kvm_host.h> #include <linux/highmem.h> #include <linux/sched/cputime.h> @@ -1864,7 +1865,8 @@ int kvm_vcpu_ioctl_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, ent->eax |=HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED ;if (evmcs_ver) ent->eax |=HV_X64_ENLIGHTENED_VMCS_RECOMMENDED ; - +if (!cpu_smt_possible ()) + ent->eax |=HV_X64_NO_NONARCH_CORESHARING ;/* * Default number of spinlock retry attempts, matches * HyperV 2016. -- 2.20.1
Vitaly Kuznetsov
maintainerRe: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Vitaly Kuznetsov
Hyper-V 2019 doesn't expose
MD_CLEARCPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.Quoted 9 lines
NoNonArchitecturalCoreSharing (CPUID
0x40000004EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).
Quoted 4 lines
That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?
(I haven't put much thought in this) but can we re-use MD_CLEAR CPUID bit for that? Like if the hypervisor can't guarantee usefulness (e.g. when two random vCPUs can be put on sibling SMT threads) of flushing, is there any reason to still make the guest think the feature is there?
Signature · 1 line
Vitaly
Paolo Bonzini
maintainerRe: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
Quoted 29 lines · Vitaly Kuznetsov
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Vitaly Kuznetsov
Hyper-V 2019 doesn't expose
MD_CLEARCPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.Quoted 9 lines
NoNonArchitecturalCoreSharing (CPUID
0x40000004EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).Quoted 4 lines
That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?
Quoted 5 lines
(I haven't put much thought in this) but can we re-use
MD_CLEARCPUID bit for that? Like if the hypervisor can't guarantee usefulness (e.g. when two random vCPUs can be put on sibling SMT threads) of flushing, is there any reason to still make the guest think the feature is there?
Yes, that's a good idea.
Paolo
Peter Zijlstra
maintainerRe: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
In reply to Vitaly KuznetsovQuoted 20 lines · Vitaly Kuznetsov
Hyper-V 2019 doesn't expose
MD_CLEARCPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.NoNonArchitecturalCoreSharing (CPUID
0x40000004EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).
Quoted 4 lines
This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).
This is purely about guest<->guest MDS, right? Ie. not worse than actual hardware.
Paolo Bonzini
maintainerRe: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines
This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).
This is purely about guest<->guest MDS, right? Ie. not worse than actual hardware.
Even within the same guest. If vCPU 1 is on virtual core 1 and vCPU 2 is on virtual core 2, but they can share the same physical core, core scheduling in the guest can do nothing about it.
Paolo
Vitaly Kuznetsov
maintainerPATCH 3/3 KVM: selftests: hyperv_cpuid: add check for NoNonArchitecturalCoreSharing bit
In reply to Vitaly KuznetsovThe bit is supposed to be '1' when SMT is not supported or forcefully disabled and '0' otherwise.
--- .../selftests/kvm/x86_64/hyperv_cpuid.c | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+)
tools/testing/selftests/kvm/x86_64/hyperv_cpuid.c +27 -0
@@ -26,6 +26,25 @@ static void guest_code(void) { } +static int smt_possible (void ) +{ +char buf[16 ]; +FILE *f; +bool res =1 ; + + f =fopen ("/sys/devices/system/cpu/smt/control" ,"r" ); +if (f) { +if (fread (buf,sizeof (*buf),sizeof (buf), f) >0 ) { +if (!strncmp (buf,"forceoff" ,8 ) || + !strncmp (buf,"notsupported" ,12 )) + res =0 ; + } +fclose (f); + } + +return res; +} +static void test_hv_cpuid (struct kvm_cpuid2 *hv_cpuid_entries,int evmcs_enabled) {
@@ -59,6 +78,14 @@ static void test_hv_cpuid(struct kvm_cpuid2 *hv_cpuid_entries,TEST_ASSERT (!entry->padding[0 ] && !entry->padding[1 ] && !entry->padding[2 ],"padding should be zero" ); +if (entry->function ==0x40000004 ) { +int nononarchcs = !!(entry->eax & (1UL <<18 )); + +TEST_ASSERT (nononarchcs == !smt_possible (), +"NoNonArchitecturalCoreSharing bit" +" doesn't reflect SMT setting" ); + } +/* * If needed for debug: * fprintf(stdout,
Signature · 1 line
2.20.1