Skip to contentSkip to conversation

ARM: KVM: allow VMs with more than 4 CPUs

PATCH New Marc Zyngier 2 participants 6 messages Oct 18, 2013 – Oct 22, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover ARM: KVM: allow VMs with more than 4 CPUs

Currently, KVM/ARM is limited to 4 A7 or A15 per VM. This is due to the fact that KVM doesn't know about clusters, and puts all CPUs at affinity level 0.

This patch series removes this limitation by allowing additionnal CPUs to be part of a separate cluster (affinity level 1).

This also requires a patch to kvmtool so the generated DT matches the expectations of the guest (posted separately).

Tested on TC2 and arm64 model.

Patches based on Christoffer's next tree as of today.

Marc Zyngier (4):
arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU
ARM: KVM: Fix MPIDR computing to support virtual clusters
ARM: KVM: fix L2CTLR to be per-cluster
ARM: KVM: drop limitation to 4 CPU VMs

 arch/arm/include/asm/kvm_emulate.h   |  5 +++++
 arch/arm/kvm/coproc.c                | 14 ++++++++++----
 arch/arm/kvm/psci.c                  | 17 +++++++++++++----
 arch/arm/kvm/reset.c                 |  4 ----
 arch/arm64/include/asm/kvm_emulate.h |  5 +++++
 5 files changed, 33 insertions(+), 12 deletions(-)
Signature · 1 line
1.8.2.3
#1 lore raw

PATCH 1/4 arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU

The KVM PSCI code blindly assumes that vcpu_id and MPIDR are the same thing. This is true when vcpus are organized as a flat topology, but is wrong when trying to emulate any other topology (such as A15 clusters).

Change the KVM PSCI CPU_ON code to look at the MPIDR instead of the vcpu_id to pick a target CPU.

---
 arch/arm/include/asm/kvm_emulate.h   |  5 +++++
 arch/arm/kvm/psci.c                  | 17 +++++++++++++----
 arch/arm64/include/asm/kvm_emulate.h |  5 +++++
 3 files changed, 23 insertions(+), 4 deletions(-)
arch/arm/include/asm/kvm_emulate.h +5 -0
index a464e8d..708e4d8 100644
@@ -157,4 +157,9 @@ static inline u32 kvm_vcpu_hvc_get_imm(struct kvm_vcpu *vcpu)
 	return kvm_vcpu_get_hsr(vcpu) & HSR_HVC_IMM_MASK;
 }
 
+static inline unsigned long kvm_vcpu_get_mpidr(struct kvm_vcpu *vcpu)
+{
+	return vcpu->arch.cp15[c0_MPIDR];
+}
+
 #endif /* __ARM_KVM_EMULATE_H__ */
arch/arm/kvm/psci.c +13 -4
index 86a693a..3112631 100644
@@ -18,6 +18,7 @@
 #include <linux/kvm_host.h>
 #include <linux/wait.h>
 
+#include <asm/cputype.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_psci.h>
 
@@ -34,22 +35,30 @@ static void kvm_psci_vcpu_off(struct kvm_vcpu *vcpu)
 static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu)
 {
 	struct kvm *kvm = source_vcpu->kvm;
-	struct kvm_vcpu *vcpu;
+	struct kvm_vcpu *vcpu = NULL, *tmp;
 	wait_queue_head_t *wq;
 	unsigned long cpu_id;
+	unsigned long mpidr;
 	phys_addr_t target_pc;
+	int i;
 
 	cpu_id = *vcpu_reg(source_vcpu, 1);
 	if (vcpu_mode_is_32bit(source_vcpu))
 		cpu_id &= ~((u32) 0);
 
-	if (cpu_id >= atomic_read(&kvm->online_vcpus))
+	kvm_for_each_vcpu(i, tmp, kvm) {
+		mpidr = kvm_vcpu_get_mpidr(tmp);
+		if ((mpidr & MPIDR_HWID_BITMASK) == (cpu_id & MPIDR_HWID_BITMASK)) {
+			vcpu = tmp;
+			break;
+		}
+	}
+
+	if (!vcpu)
 		return KVM_PSCI_RET_INVAL;
 
 	target_pc = *vcpu_reg(source_vcpu, 2);
 
-	vcpu = kvm_get_vcpu(kvm, cpu_id);
-
 	wq = kvm_arch_vcpu_wq(vcpu);
 	if (!waitqueue_active(wq))
 		return KVM_PSCI_RET_INVAL;
arch/arm64/include/asm/kvm_emulate.h +5 -0
index eec0738..6df93cd 100644
@@ -177,4 +177,9 @@ static inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu)
 	return kvm_vcpu_get_hsr(vcpu) & ESR_EL2_FSC_TYPE;
 }
 
+static inline unsigned long kvm_vcpu_get_mpidr(struct kvm_vcpu *vcpu)
+{
+	return vcpu_sys_reg(vcpu, MPIDR_EL1);
+}
+
 #endif /* __ARM64_KVM_EMULATE_H__ */
Signature · 1 line
1.8.2.3
#2 lore raw

PATCH 2/4 ARM: KVM: Fix MPIDR computing to support virtual clusters

In reply to Marc Zyngier

In order to be able to support more than 4 A7 or A15 CPUs, we need to fix the MPIDR computing to reflect the fact that both A15 and A7 can only exist in clusters of at most 4 CPUs.

Fix the MPIDR computing to allow virtual clusters to be exposed to the guest.

---
 arch/arm/kvm/coproc.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
arch/arm/kvm/coproc.c +6 -4
index a629f2c..631e6bd 100644
@@ -74,11 +74,13 @@ int kvm_handle_cp14_access(struct kvm_vcpu *vcpu, struct kvm_run *run)
 static void reset_mpidr(struct kvm_vcpu *vcpu, const struct coproc_reg *r)
 {
 	/*
-	 * Compute guest MPIDR. No need to mess around with different clusters
-	 * but we read the 'U' bit from the underlying hardware directly.
+	 * Compute guest MPIDR. We build a virtual cluster out of the
+	 * vcpu_id, but we read the 'U' bit from the underlying
+	 * hardware directly.
 	 */
-	vcpu->arch.cp15[c0_MPIDR] = (read_cpuid_mpidr() & MPIDR_SMP_BITMASK)
-					| vcpu->vcpu_id;
+	vcpu->arch.cp15[c0_MPIDR] = ((read_cpuid_mpidr() & MPIDR_SMP_BITMASK) |
+				     ((vcpu->vcpu_id >> 2) << MPIDR_LEVEL_BITS) |
+				     (vcpu->vcpu_id & 3));
 }
 
 /* TRM entries A7:4.3.31 A15:4.3.28 - RO WI */
Signature · 1 line
1.8.2.3
#3 lore raw

PATCH 3/4 ARM: KVM: fix L2CTLR to be per-cluster

In reply to Marc Zyngier

The L2CTLR register contains the number of CPUs in this cluster.

Make sure the register content is actually relevant to the vcpu that is being configured by computing the number of cores that are part of its cluster.

---
 arch/arm/kvm/coproc.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm/kvm/coproc.c +4 -0
index 631e6bd..78c0885 100644
@@ -124,6 +124,10 @@ static void reset_l2ctlr(struct kvm_vcpu *vcpu, const struct coproc_reg *r)
 	asm volatile("mrc p15, 1, %0, c9, c0, 2\n" : "=r" (l2ctlr));
 	l2ctlr &= ~(3 << 24);
 	ncores = atomic_read(&vcpu->kvm->online_vcpus) - 1;
+	/* How many cores in the current cluster and the next ones */
+	ncores -= (vcpu->vcpu_id & ~3);
+	/* Cap it to the maximum number of cores in a single cluster */
+	ncores = min(ncores, 3U);
 	l2ctlr |= (ncores & 3) << 24;
 
 	vcpu->arch.cp15[c9_L2CTLR] = l2ctlr;
Signature · 1 line
1.8.2.3
#4 lore raw

PATCH 4/4 ARM: KVM: drop limitation to 4 CPU VMs

In reply to Marc Zyngier

Now that the KVM/arm code knows about affinity, remove the hard limit of 4 vcpus per VM.

---
 arch/arm/kvm/reset.c | 4 ----
 1 file changed, 4 deletions(-)
arch/arm/kvm/reset.c +0 -4
index d9bbd83..2c5add0 100644
@@ -33,8 +33,6 @@
  * Cortex-A15 and Cortex-A7 Reset Values
  */
 
-static const int cortexa_max_cpu_idx = 3;
-
 static struct kvm_regs cortexa_regs_reset = {
 	.usr_regs.ARM_cpsr = SVC_MODE | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT,
 };
@@ -64,8 +62,6 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	switch (vcpu->arch.target) {
 	case KVM_ARM_TARGET_CORTEX_A7:
 	case KVM_ARM_TARGET_CORTEX_A15:
-		if (vcpu->vcpu_id > cortexa_max_cpu_idx)
-			return -EINVAL;
 		cpu_reset = &cortexa_regs_reset;
 		vcpu->arch.midr = read_cpuid_id();
 		cpu_vtimer_irq = &cortexa_vtimer_irq;
Signature · 1 line
1.8.2.3
#5 lore raw

cover ARM: KVM: allow VMs with more than 4 CPUs

In reply to Marc Zyngier
Quoted 20 lines · Marc Zyngier

Currently, KVM/ARM is limited to 4 A7 or A15 per VM. This is due to the fact that KVM doesn't know about clusters, and puts all CPUs at affinity level 0.

This patch series removes this limitation by allowing additionnal CPUs to be part of a separate cluster (affinity level 1).

This also requires a patch to kvmtool so the generated DT matches the expectations of the guest (posted separately).

Tested on TC2 and arm64 model.

Patches based on Christoffer's next tree as of today.

Marc Zyngier (4):
arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU
ARM: KVM: Fix MPIDR computing to support virtual clusters
ARM: KVM: fix L2CTLR to be per-cluster
ARM: KVM: drop limitation to 4 CPU VMs

Quoted 11 lines
 arch/arm/include/asm/kvm_emulate.h   |  5 +++++
 arch/arm/kvm/coproc.c                | 14 ++++++++++----
 arch/arm/kvm/psci.c                  | 17 +++++++++++++----
 arch/arm/kvm/reset.c                 |  4 ----
 arch/arm64/include/asm/kvm_emulate.h |  5 +++++
 5 files changed, 33 insertions(+), 12 deletions(-)
Signature · 1 line
1.8.2.3

ack on the series,

I'll apply these to my tree.

Signature · 1 line
Christoffer