ARM: KVM: allow VMs with more than 4 CPUs
Subsystems (3)
cover ARM: KVM: allow VMs with more than 4 CPUs
Currently, KVM/ARM is limited to 4 A7 or A15 per VM. This is due to the fact that KVM doesn't know about clusters, and puts all CPUs at affinity level 0.
This patch series removes this limitation by allowing additionnal CPUs to be part of a separate cluster (affinity level 1).
This also requires a patch to kvmtool so the generated DT matches the expectations of the guest (posted separately).
Tested on TC2 and arm64 model.
Patches based on Christoffer's next tree as of today.
Marc Zyngier (4):
arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU
ARM: KVM: Fix MPIDR computing to support virtual clusters
ARM: KVM: fix L2CTLR to be per-cluster
ARM: KVM: drop limitation to 4 CPU VMs
arch/arm/include/asm/kvm_emulate.h | 5 +++++ arch/arm/kvm/coproc.c | 14 ++++++++++---- arch/arm/kvm/psci.c | 17 +++++++++++++---- arch/arm/kvm/reset.c | 4 ---- arch/arm64/include/asm/kvm_emulate.h | 5 +++++ 5 files changed, 33 insertions(+), 12 deletions(-)
Signature · 1 line
1.8.2.3
PATCH 1/4 arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU
The KVM PSCI code blindly assumes that vcpu_id and MPIDR are the same thing. This is true when vcpus are organized as a flat topology, but is wrong when trying to emulate any other topology (such as A15 clusters).
Change the KVM PSCI CPU_ON code to look at the MPIDR instead of the vcpu_id to pick a target CPU.
--- arch/arm/include/asm/kvm_emulate.h | 5 +++++ arch/arm/kvm/psci.c | 17 +++++++++++++---- arch/arm64/include/asm/kvm_emulate.h | 5 +++++ 3 files changed, 23 insertions(+), 4 deletions(-)
arch/arm/include/asm/kvm_emulate.h +5 -0
@@ -157,4 +157,9 @@ static inline u32 kvm_vcpu_hvc_get_imm(struct kvm_vcpu *vcpu) return kvm_vcpu_get_hsr(vcpu) & HSR_HVC_IMM_MASK; } +static inline unsigned long kvm_vcpu_get_mpidr (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.cp15[c0_MPIDR]; +} +#endif /* __ARM_KVM_EMULATE_H__ */
arch/arm/kvm/psci.c +13 -4
@@ -18,6 +18,7 @@#include <linux/kvm_host.h> #include <linux/wait.h> +#include <asm/cputype.h> #include <asm/kvm_emulate.h> #include <asm/kvm_psci.h>
@@ -34,22 +35,30 @@ static void kvm_psci_vcpu_off(struct kvm_vcpu *vcpu)static unsigned long kvm_psci_vcpu_on (struct kvm_vcpu *source_vcpu) {struct kvm *kvm = source_vcpu->kvm; -struct kvm_vcpu *vcpu; +struct kvm_vcpu *vcpu =NULL , *tmp;wait_queue_head_t *wq;unsigned long cpu_id; +unsigned long mpidr;phys_addr_t target_pc; +int i; cpu_id = *vcpu_reg (source_vcpu,1 );if (vcpu_mode_is_32bit (source_vcpu)) cpu_id &= ~((u32 )0 ); -if (cpu_id >=atomic_read (&kvm->online_vcpus)) +kvm_for_each_vcpu (i, tmp, kvm) { + mpidr =kvm_vcpu_get_mpidr (tmp); +if ((mpidr &MPIDR_HWID_BITMASK ) == (cpu_id &MPIDR_HWID_BITMASK )) { + vcpu = tmp; +break ; + } + } + +if (!vcpu)return KVM_PSCI_RET_INVAL ; target_pc = *vcpu_reg (source_vcpu,2 ); - vcpu =kvm_get_vcpu (kvm, cpu_id); - wq =kvm_arch_vcpu_wq (vcpu);if (!waitqueue_active (wq))return KVM_PSCI_RET_INVAL ;
arch/arm64/include/asm/kvm_emulate.h +5 -0
@@ -177,4 +177,9 @@ static inline u8 kvm_vcpu_trap_get_fault(const struct kvm_vcpu *vcpu) return kvm_vcpu_get_hsr(vcpu) & ESR_EL2_FSC_TYPE; } +static inline unsigned long kvm_vcpu_get_mpidr (struct kvm_vcpu *vcpu) +{ +return vcpu_sys_reg (vcpu,MPIDR_EL1 ); +} +#endif /* __ARM64_KVM_EMULATE_H__ */
Signature · 1 line
1.8.2.3
PATCH 2/4 ARM: KVM: Fix MPIDR computing to support virtual clusters
In reply to Marc ZyngierIn order to be able to support more than 4 A7 or A15 CPUs, we need to fix the MPIDR computing to reflect the fact that both A15 and A7 can only exist in clusters of at most 4 CPUs.
Fix the MPIDR computing to allow virtual clusters to be exposed to the guest.
--- arch/arm/kvm/coproc.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
arch/arm/kvm/coproc.c +6 -4
@@ -74,11 +74,13 @@ int kvm_handle_cp14_access(struct kvm_vcpu *vcpu, struct kvm_run *run)static void reset_mpidr (struct kvm_vcpu *vcpu,const struct coproc_reg *r) {/* -* Compute guest MPIDR. No need to mess around with different clusters -* but we read the 'U' bit from the underlying hardware directly. +* Compute guest MPIDR. We build a virtual cluster out of the +* vcpu_id, but we read the 'U' bit from the underlying +* hardware directly. */ - vcpu->arch.cp15[c0_MPIDR] = (read_cpuid_mpidr () &MPIDR_SMP_BITMASK ) - | vcpu->vcpu_id; + vcpu->arch.cp15[c0_MPIDR] = ((read_cpuid_mpidr () &MPIDR_SMP_BITMASK ) | + ((vcpu->vcpu_id >>2 ) <<MPIDR_LEVEL_BITS ) | + (vcpu->vcpu_id &3 )); }/* TRM entries A7:4.3.31 A15:4.3.28 - RO WI */
Signature · 1 line
1.8.2.3
PATCH 3/4 ARM: KVM: fix L2CTLR to be per-cluster
In reply to Marc ZyngierThe L2CTLR register contains the number of CPUs in this cluster.
Make sure the register content is actually relevant to the vcpu that is being configured by computing the number of cores that are part of its cluster.
--- arch/arm/kvm/coproc.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm/kvm/coproc.c +4 -0
@@ -124,6 +124,10 @@ static void reset_l2ctlr(struct kvm_vcpu *vcpu, const struct coproc_reg *r) asmvolatile ("mrc p15, 1, %0, c9, c0, 2\n" :"=r" (l2ctlr)); l2ctlr &= ~(3 <<24 ); ncores =atomic_read (&vcpu->kvm->online_vcpus) -1 ; +/* How many cores in the current cluster and the next ones */ + ncores -= (vcpu->vcpu_id & ~3 ); +/* Cap it to the maximum number of cores in a single cluster */ + ncores =min (ncores,3U ); l2ctlr |= (ncores &3 ) <<24 ; vcpu->arch.cp15[c9_L2CTLR] = l2ctlr;
Signature · 1 line
1.8.2.3
PATCH 4/4 ARM: KVM: drop limitation to 4 CPU VMs
In reply to Marc ZyngierNow that the KVM/arm code knows about affinity, remove the hard limit of 4 vcpus per VM.
--- arch/arm/kvm/reset.c | 4 ---- 1 file changed, 4 deletions(-)
arch/arm/kvm/reset.c +0 -4
@@ -33,8 +33,6 @@* Cortex-A15 and Cortex-A7 Reset Values */ -static const int cortexa_max_cpu_idx =3 ; -static struct kvm_regs cortexa_regs_reset = { .usr_regs.ARM_cpsr =SVC_MODE |PSR_A_BIT |PSR_I_BIT |PSR_F_BIT , };
@@ -64,8 +62,6 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)switch (vcpu->arch.target) {case KVM_ARM_TARGET_CORTEX_A7 :case KVM_ARM_TARGET_CORTEX_A15 : -if (vcpu->vcpu_id > cortexa_max_cpu_idx) -return -EINVAL ; cpu_reset = &cortexa_regs_reset; vcpu->arch.midr =read_cpuid_id (); cpu_vtimer_irq = &cortexa_vtimer_irq;
Signature · 1 line
1.8.2.3
cover ARM: KVM: allow VMs with more than 4 CPUs
In reply to Marc ZyngierQuoted 20 lines · Marc Zyngier
Currently, KVM/ARM is limited to 4 A7 or A15 per VM. This is due to the fact that KVM doesn't know about clusters, and puts all CPUs at affinity level 0.
This patch series removes this limitation by allowing additionnal CPUs to be part of a separate cluster (affinity level 1).
This also requires a patch to kvmtool so the generated DT matches the expectations of the guest (posted separately).
Tested on TC2 and arm64 model.
Patches based on Christoffer's next tree as of today.
Marc Zyngier (4):
arm/arm64: KVM: PSCI: use MPIDR to identify a target CPU
ARM: KVM: Fix MPIDR computing to support virtual clusters
ARM: KVM: fix L2CTLR to be per-cluster
ARM: KVM: drop limitation to 4 CPU VMs
Quoted 11 lines
arch/arm/include/asm/kvm_emulate.h | 5 +++++ arch/arm/kvm/coproc.c | 14 ++++++++++---- arch/arm/kvm/psci.c | 17 +++++++++++++---- arch/arm/kvm/reset.c | 4 ---- arch/arm64/include/asm/kvm_emulate.h | 5 +++++ 5 files changed, 33 insertions(+), 12 deletions(-)Signature · 1 line
1.8.2.3
ack on the series,
I'll apply these to my tree.
Signature · 1 line
Christoffer