Skip to contentSkip to conversation

KVM: SVM: Inhibit APIC virtualization for X2APIC guest

PATCH In review Oliver Upton 5 participants 8 messages Feb 28, 2020 – Mar 2, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 KVM: SVM: Inhibit APIC virtualization for X2APIC guest

The AVIC does not support guest use of the x2APIC interface. Currently, KVM simply chooses to squash the x2APIC feature in the guest's CPUID If the AVIC is enabled. Doing so prevents KVM from running a guest with greater than 255 vCPUs, as such a guest necessitates the use of the x2APIC interface.

Instead, inhibit AVIC enablement on a per-VM basis whenever the x2APIC feature is set in the guest's CPUID.

---

Parent commit: a93236fcbe1d ("KVM: s390: rstify new ioctls in api.rst")

v1 => v2:
- Adopt Paolo's suggestion to inhibit the AVIC by default rather than requiring opt-in
- Drop now unnecessary module parameter

 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/svm.c              | 15 +++++++++------
 2 files changed, 10 insertions(+), 6 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 98959e8cd448..9d40132a3ae2 100644
@@ -890,6 +890,7 @@ enum kvm_irqchip_mode {
 #define APICV_INHIBIT_REASON_NESTED     2
 #define APICV_INHIBIT_REASON_IRQWIN     3
 #define APICV_INHIBIT_REASON_PIT_REINJ  4
+#define APICV_INHIBIT_REASON_X2APIC	5
 
 struct kvm_arch {
 	unsigned long n_used_mmu_pages;
arch/x86/kvm/svm.c +9 -6
index ad3f5b178a03..b82a500bccb7 100644
@@ -6027,7 +6027,13 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu)
 	if (!kvm_vcpu_apicv_active(vcpu))
 		return;
 
-	guest_cpuid_clear(vcpu, X86_FEATURE_X2APIC);
+	/*
+	 * AVIC does not work with an x2APIC mode guest. If the X2APIC feature
+	 * is exposed to the guest, disable AVIC.
+	 */
+	if (guest_cpuid_has(vcpu, X86_FEATURE_X2APIC))
+		kvm_request_apicv_update(vcpu->kvm, false,
+					 APICV_INHIBIT_REASON_X2APIC);
 
 	/*
 	 * Currently, AVIC does not work with nested virtualization.
@@ -6043,10 +6049,6 @@ static void svm_cpuid_update(struct kvm_vcpu *vcpu)
 static void svm_set_supported_cpuid(u32 func, struct kvm_cpuid_entry2 *entry)
 {
 	switch (func) {
-	case 0x1:
-		if (avic)
-			entry->ecx &= ~F(X2APIC);
-		break;
 	case 0x80000001:
 		if (nested)
 			entry->ecx |= (1 << 2); /* Set SVM bit */
@@ -7370,7 +7372,8 @@ static bool svm_check_apicv_inhibit_reasons(ulong bit)
 			  BIT(APICV_INHIBIT_REASON_HYPERV) |
 			  BIT(APICV_INHIBIT_REASON_NESTED) |
 			  BIT(APICV_INHIBIT_REASON_IRQWIN) |
-			  BIT(APICV_INHIBIT_REASON_PIT_REINJ);
+			  BIT(APICV_INHIBIT_REASON_PIT_REINJ) |
+			  BIT(APICV_INHIBIT_REASON_X2APIC);
 
 	return supported & BIT(bit);
 }
Signature · 1 line
2.25.1.481.gfbce0eb801-goog
#1 lore raw

PATCH v2 2/2 KVM: SVM: Enable AVIC by default

Switch the default value of the 'avic' module parameter to 1.

---
 arch/x86/kvm/svm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm.c +1 -1
index b82a500bccb7..70d2df13eb02 100644
@@ -358,7 +358,7 @@ static int nested = true;
 module_param(nested, int, S_IRUGO);
 
 /* enable / disable AVIC */
-static int avic;
+static int avic = 1;
 #ifdef CONFIG_X86_LOCAL_APIC
 module_param(avic, int, S_IRUGO);
 #endif
Signature · 1 line
2.25.1.481.gfbce0eb801-goog
#2 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

Quoted 23 lines · Oliver Upton

Switch the default value of the 'avic' module parameter to 1.

---
 arch/x86/kvm/svm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm.c +1 -2
index b82a500bccb7..70d2df13eb02 100644
@@ -358,7 +358,7 @@ static int nested = true;
 module_param(nested, int, S_IRUGO);
 
 /* enable / disable AVIC */
-static int avic;
+static int avic = 1;
 #ifdef CONFIG_X86_LOCAL_APIC
 module_param(avic, int, S_IRUGO);
 #endif
--
Quoted 1 line

2.25.1.481.gfbce0eb801-goog

How extensively has this been tested? Why hasn't someone from AMD suggested this change?

#3 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

Quoted 17 lines · Jim Mattson
Quoted 16 lines · Oliver Upton

Switch the default value of the 'avic' module parameter to 1.

---
 arch/x86/kvm/svm.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/svm.c +0 -0
index b82a500bccb7..70d2df13eb02 100644
@@ -358,7 +358,7 @@ static int nested = true;
 module_param(nested, int, S_IRUGO);
 
Quoted 11 lines
Quoted 8 lines
arch/x86/kvm/svm.c +1 -2
 /* enable / disable AVIC */
-static int avic;
+static int avic = 1;
 #ifdef CONFIG_X86_LOCAL_APIC
 module_param(avic, int, S_IRUGO);
 #endif
--

2.25.1.481.gfbce0eb801-goog

How extensively has this been tested? Why hasn't someone from AMD suggested this change?

I personally don't suggest enable AVIC by default. There are cases of slow AVIC doorbell delivery, due to delivery path and contention under a large number of guest cores.

Quoted 1 line
maintainer
#4 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

Quoted 6 lines · Wei Huang
Quoted 2 lines

How extensively has this been tested? Why hasn't someone from AMD suggested this change?

I personally don't suggest enable AVIC by default. There are cases of slow AVIC doorbell delivery, due to delivery path and contention under a large number of guest cores.

To clarify, this is a hardware issue, right?

Note that in practice this patch series wouldn't change much, because x2apic is enabled by default by userspace (it has better performance than memory-mapped APIC) and patch 1 in turn inhibits APICv if the guest has the X2APIC CPUID bit set.

Paolo

#5 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

Quoted 8 lines · Paolo Bonzini
Quoted 6 lines · Wei Huang
Quoted 2 lines

How extensively has this been tested? Why hasn't someone from AMD suggested this change?

I personally don't suggest enable AVIC by default. There are cases of slow AVIC doorbell delivery, due to delivery path and contention under a large number of guest cores.

Quoted 6 lines

To clarify, this is a hardware issue, right?

Note that in practice this patch series wouldn't change much, because x2apic is enabled by default by userspace (it has better performance than memory-mapped APIC) and patch 1 in turn inhibits APICv if the guest has the X2APIC CPUID bit set.

QEMU will work fine with this option ON due to x2APIC, just like what you said. If you feel other emulators using KVM will behave similarly, I can revoke my concern.

Quoted 3 lines

Paolo

maintainer
#6 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

Quoted 5 lines · Wei Huang
Quoted 5 lines
Quoted 3 lines

I personally don't suggest enable AVIC by default. There are cases of slow AVIC doorbell delivery, due to delivery path and contention under a large number of guest cores.

To clarify, this is a hardware issue, right?

Quoted 7 lines
Quoted 4 lines

Note that in practice this patch series wouldn't change much, because x2apic is enabled by default by userspace (it has better performance than memory-mapped APIC) and patch 1 in turn inhibits APICv if the guest has the X2APIC CPUID bit set.

QEMU will work fine with this option ON due to x2APIC, just like what you said. If you feel other emulators using KVM will behave similarly, I can revoke my concern.

No, it's okay for now to leave it disabled. It would be nice if this information would be more easily available than by you lurking on the mailing list. :)

Paolo

#7 lore raw

Re: [PATCH v2 2/2] KVM: SVM: Enable AVIC by default

In reply to Wei Huang
Quoted 3 lines · Wei Huang

I personally don't suggest enable AVIC by default. There are cases of slow AVIC doorbell delivery, due to delivery path and contention under a large number of guest cores.

Under what conditions would you suggest enabling AVIC?