KVM: arm64: Ensure TLBs and I-cache are private to each vCPU in pKVM

All vCPUs of a VM share its VMID, so KVM flushes a physical CPU's TLBs and I-cache for the VM whenever a different vCPU of that VM is loaded there. pKVM guests, protected or not, run on the hypervisor's stage-2 and VMID, and the host doesn't flush for them. Nothing at EL2 flushes instead, so a vCPU can hit TLB and I-cache entries that another vCPU of the same VM left on that CPU.

Track at EL2 the last vCPU of each VM loaded on each CPU, and flush when it changes. EL2 can't allocate per-CPU memory for each VM, as the host does, so use one per-CPU array indexed by the VM's table entry. Reset the entry's slot on every CPU when a VM reserves it, so that a new VM starts with no last vCPU, whether the entry is unused or was left by a VM that was torn down.

Fixes fce886a602073 ("KVM: arm64: Plumb the pKVM MMU in KVM")

Signed-off-by Fuad Tabba <[email protected]>

---
Changes since v1 [1]:
- Dropped patch 1, which added a CPU number at EL2. The last vCPU is now tracked in a per-CPU array indexed by VM table entry. (Marc)
- Moved the check from __pkvm_vcpu_load() to pkvm_load_hyp_vcpu().
- The array is unsigned int, initialised to UINT_MAX. (Vincent)
- Dropped Joey's Reviewed-by, since the patch changed.

Based on kvmarm/next (dd0bc00e3e920).

[1] https://lore.kernel.org/all/[email protected]/

 arch/arm64/kvm/hyp/nvhe/pkvm.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

arch/arm64/kvm/hyp/nvhe/pkvm.c +20 -2

index ac020d0501d2e..1fc59229c8e86 100644
@@ -38,6 +38,9 @@ unsigned int kvm_host_sve_max_vl;
  */
 static DEFINE_PER_CPU(struct pkvm_hyp_vcpu *, loaded_hyp_vcpu);
 
+/* Per VM table entry, the vcpu_idx of the last vCPU of that VM loaded on this CPU. */
+static DEFINE_PER_CPU(unsigned int, last_vcpu_ran[KVM_MAX_PVMS]);
+
 static void pkvm_vcpu_reset_hcr(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS;

@@ -261,6 +264,7 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
 {
 	struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
 	struct pkvm_hyp_vm *hyp_vm;
+	unsigned int *last_ran;
 
 	/* Cannot load a new vcpu without putting the old one first. */
 	if (__this_cpu_read(loaded_hyp_vcpu))

@@ -290,8 +294,18 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
 unlock:
 	hyp_spin_unlock(&vm_table_lock);
 
-	if (hyp_vcpu)
-		__this_cpu_write(loaded_hyp_vcpu, hyp_vcpu);
+	if (!hyp_vcpu)
+		return NULL;
+
+	__this_cpu_write(loaded_hyp_vcpu, hyp_vcpu);
+
+	/* TLBs and I-cache are private to each vCPU, see vcpu_prepare_mmu(). */
+	last_ran = this_cpu_ptr(&last_vcpu_ran[vm_handle_to_idx(handle)]);
+	if (*last_ran != vcpu_idx) {
+		__kvm_flush_cpu_context(hyp_vcpu->vcpu.arch.hw_mmu);
+		*last_ran = vcpu_idx;
+	}
+
 	return hyp_vcpu;
 }
 

@@ -652,6 +666,7 @@ static int find_free_vm_table_entry(void)
  */
 static int allocate_vm_table_entry(void)
 {
+	unsigned int cpu;
 	int idx;
 
 	hyp_assert_lock_held(&vm_table_lock);

@@ -670,6 +685,9 @@ static int allocate_vm_table_entry(void)
 
 	vm_table[idx] = RESERVED_ENTRY;
 
+	for (cpu = 0; cpu < hyp_nr_cpus; cpu++)
+		per_cpu(last_vcpu_ran[idx], cpu) = UINT_MAX;
+
 	return idx;
 }
 

base-commit: dd0bc00e3e9201fd49b62ff9cc07890c89003afe

Signature · 1 line

2.47.3

All vCPUs of a VM share its VMID, so KVM flushes a physical CPU's TLBs and I-cache for the VM whenever a different vCPU of that VM is loaded there. pKVM guests, protected or not, run on the hypervisor's stage-2 and VMID, and the host doesn't flush for them. Nothing at EL2 flushes instead, so a vCPU can hit TLB and I-cache entries that another vCPU of the same VM left on that CPU.

Track at EL2 the last vCPU of each VM loaded on each CPU, and flush when it changes. EL2 can't allocate per-CPU memory for each VM, as the host does, so use one per-CPU array indexed by the VM's table entry. Reset the entry's slot on every CPU when a VM reserves it, so that a new VM starts with no last vCPU, whether the entry is unused or was left by a VM that was torn down.

Fixes fce886a602073 ("KVM: arm64: Plumb the pKVM MMU in KVM")

Signed-off-by Fuad Tabba <[email protected]>

---
Changes since v1 [1]:
- Dropped patch 1, which added a CPU number at EL2. The last vCPU is now tracked in a per-CPU array indexed by VM table entry. (Marc)
- Moved the check from __pkvm_vcpu_load() to pkvm_load_hyp_vcpu().
- The array is unsigned int, initialised to UINT_MAX. (Vincent)
- Dropped Joey's Reviewed-by, since the patch changed.

Based on kvmarm/next (dd0bc00e3e920).

[1] https://lore.kernel.org/all/[email protected]/

 arch/arm64/kvm/hyp/nvhe/pkvm.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

arch/arm64/kvm/hyp/nvhe/pkvm.c +20 -2

index ac020d0501d2e..1fc59229c8e86 100644
@@ -38,6 +38,9 @@ unsigned int kvm_host_sve_max_vl;
  */
 static DEFINE_PER_CPU(struct pkvm_hyp_vcpu *, loaded_hyp_vcpu);
 
+/* Per VM table entry, the vcpu_idx of the last vCPU of that VM loaded on this CPU. */
+static DEFINE_PER_CPU(unsigned int, last_vcpu_ran[KVM_MAX_PVMS]);
+
 static void pkvm_vcpu_reset_hcr(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS;

@@ -261,6 +264,7 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
 {
 	struct pkvm_hyp_vcpu *hyp_vcpu = NULL;
 	struct pkvm_hyp_vm *hyp_vm;
+	unsigned int *last_ran;
 
 	/* Cannot load a new vcpu without putting the old one first. */
 	if (__this_cpu_read(loaded_hyp_vcpu))

@@ -290,8 +294,18 @@ struct pkvm_hyp_vcpu *pkvm_load_hyp_vcpu(pkvm_handle_t handle,
 unlock:
 	hyp_spin_unlock(&vm_table_lock);
 
-	if (hyp_vcpu)
-		__this_cpu_write(loaded_hyp_vcpu, hyp_vcpu);
+	if (!hyp_vcpu)
+		return NULL;
+
+	__this_cpu_write(loaded_hyp_vcpu, hyp_vcpu);
+
+	/* TLBs and I-cache are private to each vCPU, see vcpu_prepare_mmu(). */
+	last_ran = this_cpu_ptr(&last_vcpu_ran[vm_handle_to_idx(handle)]);
+	if (*last_ran != vcpu_idx) {
+		__kvm_flush_cpu_context(hyp_vcpu->vcpu.arch.hw_mmu);
+		*last_ran = vcpu_idx;
+	}
+
 	return hyp_vcpu;
 }
 

@@ -652,6 +666,7 @@ static int find_free_vm_table_entry(void)
  */
 static int allocate_vm_table_entry(void)
 {
+	unsigned int cpu;
 	int idx;
 
 	hyp_assert_lock_held(&vm_table_lock);

@@ -670,6 +685,9 @@ static int allocate_vm_table_entry(void)
 
 	vm_table[idx] = RESERVED_ENTRY;
 
+	for (cpu = 0; cpu < hyp_nr_cpus; cpu++)
+		per_cpu(last_vcpu_ran[idx], cpu) = UINT_MAX;
+
 	return idx;
 }
 

base-commit: dd0bc00e3e9201fd49b62ff9cc07890c89003afe

Signature · 1 line

2.47.3

All vCPUs of a VM share its VMID, so KVM flushes a physical CPU's TLBs and I-cache for the VM whenever a different vCPU of that VM is loaded there. pKVM guests, protected or not, run on the hypervisor's stage-2 and VMID, and the host doesn't flush for them. Nothing at EL2 flushes instead, so a vCPU can hit TLB and I-cache entries that another vCPU of the same VM left on that CPU.

[...]