Skip to contentSkip to conversation

KVM: x86: hyper-v: make L2 Hyper-V 2019 on KVM guests see MD_CLEAR

PATCH In review Vitaly Kuznetsov 4 participants 12 messages Sep 16, 2019 – Sep 23, 2019

Subsystems (8)
x86 KVM Kselftest KVM x86 +4 more subsystems
lore
Share & tools
lore

Vitaly Kuznetsov

author maintainer
#0 lore raw

cover KVM: x86: hyper-v: make L2 Hyper-V 2019 on KVM guests see MD_CLEAR

[The series is KVM specific but the first patch of the series likely requires
someone else's ACK. hyperv-tlfs.h gets a small addition too.]

It was discovered that L2 guests on Hyper-V 2019 on KVM don't see MD_CLEAR bit (and thus think they're MDS vulnerable) even when it is present on the host. Turns out, Hyper-V is filtering it out because it is not sure the topology L0 is exposing is trustworthy and generally it is not. In some specific cases (e.g. when SMT is unsupported or forcesully disabled) it is and we can tell this to userspace hoping that it'll pass this info to L1. See PATCH2 of the series for additional details.

The series can be tested with QEMU-4.1+ and 'hv-passthrough' CPU flag.

Vitaly Kuznetsov (3):
cpu/SMT: create and export cpu_smt_possible()
KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when
SMT is impossible
KVM: selftests: hyperv_cpuid: add check for
NoNonArchitecturalCoreSharing bit

 arch/x86/include/asm/hyperv-tlfs.h            |  7 +++++
 arch/x86/kvm/hyperv.c                         |  4 ++-
 include/linux/cpu.h                           |  2 ++
 kernel/cpu.c                                  | 11 ++++++--
 .../selftests/kvm/x86_64/hyperv_cpuid.c       | 27 +++++++++++++++++++
 5 files changed, 48 insertions(+), 3 deletions(-)
Signature · 1 line
2.20.1
author maintainer
#1 lore raw

PATCH 1/3 cpu/SMT: create and export cpu_smt_possible()

KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export cpu_smt_possible() answering this question.

---
 include/linux/cpu.h |  2 ++
 kernel/cpu.c        | 11 +++++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)
include/linux/cpu.h +2 -0
index fcb1386bb0d4..6d48fc456d58 100644
@@ -201,12 +201,14 @@ enum cpuhp_smt_control {
 extern enum cpuhp_smt_control cpu_smt_control;
 extern void cpu_smt_disable(bool force);
 extern void cpu_smt_check_topology(void);
+extern bool cpu_smt_possible(void);
 extern int cpuhp_smt_enable(void);
 extern int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval);
 #else
 # define cpu_smt_control		(CPU_SMT_NOT_IMPLEMENTED)
 static inline void cpu_smt_disable(bool force) { }
 static inline void cpu_smt_check_topology(void) { }
+static inline bool cpu_smt_possible(void) { return false; }
 static inline int cpuhp_smt_enable(void) { return 0; }
 static inline int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval) { return 0; }
 #endif
kernel/cpu.c +9 -2
index e84c0873559e..2f8c2631e641 100644
@@ -389,8 +389,7 @@ enum cpuhp_smt_control cpu_smt_control __read_mostly = CPU_SMT_ENABLED;
 
 void __init cpu_smt_disable(bool force)
 {
-	if (cpu_smt_control == CPU_SMT_FORCE_DISABLED ||
-		cpu_smt_control == CPU_SMT_NOT_SUPPORTED)
+	if (!cpu_smt_possible())
 		return;
 
 	if (force) {
@@ -435,6 +434,14 @@ static inline bool cpu_smt_allowed(unsigned int cpu)
 	 */
 	return !per_cpu(cpuhp_state, cpu).booted_once;
 }
+
+/* Returns true if SMT is not supported of forcefully (irreversibly) disabled */
+bool cpu_smt_possible(void)
+{
+	return cpu_smt_control != CPU_SMT_FORCE_DISABLED &&
+		cpu_smt_control != CPU_SMT_NOT_SUPPORTED;
+}
+EXPORT_SYMBOL_GPL(cpu_smt_possible);
 #else
 static inline bool cpu_smt_allowed(unsigned int cpu) { return true; }
 #endif
Signature · 1 line
2.20.1
#2 lore raw

Re: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()

Quoted 4 lines · Vitaly Kuznetsov

KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export cpu_smt_possible() answering this question.

It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?

maintainer
#3 lore raw

Re: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()

Quoted 10 lines · Jim Mattson
Quoted 3 lines

KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export cpu_smt_possible() answering this question.

It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?

Yes, however that scheduler API could also rely on something like cpu_smt_possible(), at least in the case where core scheduling is not active, so this is still a step in the right direction.

Paolo

author maintainer
#4 lore raw

Re: [PATCH 1/3] cpu/SMT: create and export cpu_smt_possible()

In reply to Jim Mattson
Quoted 6 lines · Jim Mattson
Quoted 4 lines · Vitaly Kuznetsov

KVM needs to know if SMT is theoretically possible, this means it is supported and not forcefully disabled ('nosmt=force'). Create and export cpu_smt_possible() answering this question.

Quoted 7 lines

It seems to me that KVM really just wants to know if the scheduler can be trusted to avoid violating the invariant expressed by the Hyper-V enlightenment, NoNonArchitecturalCoreSharing. It is possible to do that even when SMT is enabled, if the scheduler is core-aware. Wouldn't it be better to implement a scheduler API that told you exactly what you wanted to know, rather than trying to infer the answer from various breadcrumbs?

(I know not that much about scheduler so please bear with me)

Having a trustworthy scheduler not placing unrelated (not exposed as sibling SMT threads to a guest or vCPUs from different guests) on sibling SMT threads when it's not limited with affinity is definitely a good thing. We, however, also need to know if vCPU pinning is planned for the guest: like when QEMU vCPU threads are created they're not pinned, however, libvirt pins them if needed before launching the guest. So 'NoNonArchitecturalCoreSharing' can also be set in two cases:
- No vCPU pinning is planned but the scheduler is aware of the problem (I'm not sure it is nowadays)
- The upper layer promises to do the right pinning.

This patch series, however, doesn't go that deep, it only covers the simplest case: SMT is unavailable or forcefully disabled. I'll try to learn more about scheduler though.

Signature · 1 line
Vitaly
author maintainer
#5 lore raw

PATCH 2/3 KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

In reply to Vitaly Kuznetsov

Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.

NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).

This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).

---
 arch/x86/include/asm/hyperv-tlfs.h | 7 +++++++
 arch/x86/kvm/hyperv.c              | 4 +++-
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/hyperv-tlfs.h +7 -0
index af78cd72b8f3..989a1efe7f5e 100644
@@ -170,6 +170,13 @@
 /* Recommend using enlightened VMCS */
 #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED		BIT(14)
 
+/*
+ * Virtual processor will never share a physical core with another virtual
+ * processor, except for virtual processors that are reported as sibling SMT
+ * threads.
+ */
+#define HV_X64_NO_NONARCH_CORESHARING                  BIT(18)
+
 /* Nested features. These are HYPERV_CPUID_NESTED_FEATURES.EAX bits. */
 #define HV_X64_NESTED_GUEST_MAPPING_FLUSH		BIT(18)
 #define HV_X64_NESTED_MSR_BITMAP			BIT(19)
arch/x86/kvm/hyperv.c +3 -1
index fff790a3f4ee..9c187d16a9cd 100644
@@ -23,6 +23,7 @@
 #include "ioapic.h"
 #include "hyperv.h"
 
+#include <linux/cpu.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
 #include <linux/sched/cputime.h>
@@ -1864,7 +1865,8 @@ int kvm_vcpu_ioctl_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
 			ent->eax |= HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED;
 			if (evmcs_ver)
 				ent->eax |= HV_X64_ENLIGHTENED_VMCS_RECOMMENDED;
-
+			if (!cpu_smt_possible())
+				ent->eax |= HV_X64_NO_NONARCH_CORESHARING;
 			/*
 			 * Default number of spinlock retry attempts, matches
 			 * HyperV 2016.
Signature · 1 line
2.20.1
#6 lore raw

Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

Quoted 11 lines · Vitaly Kuznetsov

Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.

Quoted 9 lines

NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).

That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?

Quoted 54 lines

This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).

---
 arch/x86/include/asm/hyperv-tlfs.h | 7 +++++++
 arch/x86/kvm/hyperv.c              | 4 +++-
 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/include/asm/hyperv-tlfs.h +7 -0
index af78cd72b8f3..989a1efe7f5e 100644
@@ -170,6 +170,13 @@
 /* Recommend using enlightened VMCS */
 #define HV_X64_ENLIGHTENED_VMCS_RECOMMENDED            BIT(14)
 
+/*
+ * Virtual processor will never share a physical core with another virtual
+ * processor, except for virtual processors that are reported as sibling SMT
+ * threads.
+ */
+#define HV_X64_NO_NONARCH_CORESHARING                  BIT(18)
+
 /* Nested features. These are HYPERV_CPUID_NESTED_FEATURES.EAX bits. */
 #define HV_X64_NESTED_GUEST_MAPPING_FLUSH              BIT(18)
 #define HV_X64_NESTED_MSR_BITMAP                       BIT(19)
arch/x86/kvm/hyperv.c +3 -2
index fff790a3f4ee..9c187d16a9cd 100644
@@ -23,6 +23,7 @@
 #include "ioapic.h"
 #include "hyperv.h"
 
+#include <linux/cpu.h>
 #include <linux/kvm_host.h>
 #include <linux/highmem.h>
 #include <linux/sched/cputime.h>
@@ -1864,7 +1865,8 @@ int kvm_vcpu_ioctl_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,
                        ent->eax |= HV_X64_EX_PROCESSOR_MASKS_RECOMMENDED;
                        if (evmcs_ver)
                                ent->eax |= HV_X64_ENLIGHTENED_VMCS_RECOMMENDED;
-
+                       if (!cpu_smt_possible())
+                               ent->eax |= HV_X64_NO_NONARCH_CORESHARING;
                        /*
                         * Default number of spinlock retry attempts, matches
                         * HyperV 2016.
--

2.20.1

author maintainer
#7 lore raw

Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

Quoted 22 lines · Jim Mattson
Quoted 11 lines · Vitaly Kuznetsov

Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.

Quoted 9 lines

NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).

Quoted 4 lines

That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?

(I haven't put much thought in this) but can we re-use MD_CLEAR CPUID bit for that? Like if the hypervisor can't guarantee usefulness (e.g. when two random vCPUs can be put on sibling SMT threads) of flushing, is there any reason to still make the guest think the feature is there?

Signature · 1 line
Vitaly
maintainer
#8 lore raw

Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

Quoted 29 lines · Vitaly Kuznetsov
Quoted 22 lines · Jim Mattson
Quoted 11 lines · Vitaly Kuznetsov

Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.

Quoted 9 lines

NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).

Quoted 4 lines

That's a nice bit of information. Have you considered a mechanism for communicating this information to kvm guests in a way that doesn't require Hyper-V enlightenments?

Quoted 5 lines

(I haven't put much thought in this) but can we re-use MD_CLEAR CPUID bit for that? Like if the hypervisor can't guarantee usefulness (e.g. when two random vCPUs can be put on sibling SMT threads) of flushing, is there any reason to still make the guest think the feature is there?

Yes, that's a good idea.

Paolo

maintainer
#9 lore raw

Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

In reply to Vitaly Kuznetsov
Quoted 20 lines · Vitaly Kuznetsov

Hyper-V 2019 doesn't expose MD_CLEAR CPUID bit to guests when it cannot guarantee that two virtual processors won't end up running on sibling SMT threads without knowing about it. This is done as an optimization as in this case there is nothing the guest can do to protect itself against MDS and issuing additional flush requests is just pointless. On bare metal the topology is known, however, when Hyper-V is running nested (e.g. on top of KVM) it needs an additional piece of information: a confirmation that the exposed topology (wrt vCPU placement on different SMT threads) is trustworthy.

NoNonArchitecturalCoreSharing (CPUID 0x40000004 EAX bit 18) is described in TLFS as follows: "Indicates that a virtual processor will never share a physical core with another virtual processor, except for virtual processors that are reported as sibling SMT threads." From KVM we can give such guarantee in two cases:
- SMT is unsupported or forcefully disabled (just 'disabled' doesn't work as it can become re-enabled during the lifetime of the guest).
- vCPUs are properly pinned so the scheduler won't put them on sibling SMT threads (when they're not reported as such).

Quoted 4 lines

This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).

This is purely about guest<->guest MDS, right? Ie. not worse than actual hardware.

maintainer
#10 lore raw

Re: [PATCH 2/3] KVM: x86: hyper-v: set NoNonArchitecturalCoreSharing CPUID bit when SMT is impossible

Quoted 6 lines · Peter Zijlstra
Quoted 4 lines

This patch reports NoNonArchitecturalCoreSharing bit in to userspace in the first case. The second case is outside of KVM's domain of responsibility (as vCPU pinning is actually done by someone who manages KVM's userspace - e.g. libvirt pinning QEMU threads).

This is purely about guest<->guest MDS, right? Ie. not worse than actual hardware.

Even within the same guest. If vCPU 1 is on virtual core 1 and vCPU 2 is on virtual core 2, but they can share the same physical core, core scheduling in the guest can do nothing about it.

Paolo

author maintainer
#11 lore raw

PATCH 3/3 KVM: selftests: hyperv_cpuid: add check for NoNonArchitecturalCoreSharing bit

In reply to Vitaly Kuznetsov

The bit is supposed to be '1' when SMT is not supported or forcefully disabled and '0' otherwise.

---
 .../selftests/kvm/x86_64/hyperv_cpuid.c       | 27 +++++++++++++++++++
 1 file changed, 27 insertions(+)
tools/testing/selftests/kvm/x86_64/hyperv_cpuid.c +27 -0
index ee59831fbc98..443a2b54645b 100644
@@ -26,6 +26,25 @@ static void guest_code(void)
 {
 }
 
+static int smt_possible(void)
+{
+	char buf[16];
+	FILE *f;
+	bool res = 1;
+
+	f = fopen("/sys/devices/system/cpu/smt/control", "r");
+	if (f) {
+		if (fread(buf, sizeof(*buf), sizeof(buf), f) > 0) {
+			if (!strncmp(buf, "forceoff", 8) ||
+			    !strncmp(buf, "notsupported", 12))
+				res = 0;
+		}
+		fclose(f);
+	}
+
+	return res;
+}
+
 static void test_hv_cpuid(struct kvm_cpuid2 *hv_cpuid_entries,
 			  int evmcs_enabled)
 {
@@ -59,6 +78,14 @@ static void test_hv_cpuid(struct kvm_cpuid2 *hv_cpuid_entries,
 		TEST_ASSERT(!entry->padding[0] && !entry->padding[1] &&
 			    !entry->padding[2], "padding should be zero");
 
+		if (entry->function == 0x40000004) {
+			int nononarchcs = !!(entry->eax & (1UL << 18));
+
+			TEST_ASSERT(nononarchcs == !smt_possible(),
+				    "NoNonArchitecturalCoreSharing bit"
+				    " doesn't reflect SMT setting");
+		}
+
 		/*
 		 * If needed for debug:
 		 * fprintf(stdout,
Signature · 1 line
2.20.1