KVM: arm64: Fix teardown of non-protected VMs with pKVM

Hi folks,

Sashiko continues to throw rocks at me.

This time, it spotted that my fix (introduced in v5 [1] of the pKVM series) to prevent taking a reference on a VM in the 'is_dying' state also prevents unsharing of pages shared with a non-protected VM if that VM is torn down by its VM fd being destroyed, rather than the usual path via the MMU notifiers.

Rather than send a v6 of the whole series, here are three patches that apply on top of v5 and fix the issue by (a) preventing teardown of a referenced VM and (b) allowing some references to be taken on a dying VM. As an added bonus, this simplifies the locking on the reclaim path because now a VM reference is enough to stop the page-tables from going away.

Cheers,

Will

[1] https://lore.kernel.org/kvmarm/[email protected]

--->8

Will Deacon (3):
KVM: arm64: Prevent teardown finalisation of referenced 'hyp_vm'
KVM: arm64: Allow get_pkvm_hyp_vm() to take a reference to a dying VM
KVM: arm64: Don't hold 'vm_table_lock' across guest page reclaim

 arch/arm64/kvm/hyp/nvhe/pkvm.c | 53 +++++++++++++++++-----------------
 1 file changed, 26 insertions(+), 27 deletions(-)

2.53.0.1118.gaef5881109-goog

Destroying a 'hyp_vm' with an elevated referenced count in __pkvm_finalize_teardown_vm() is only going to lead to tears.

In preparation for allowing limited references to be acquired on dying VMs during the teardown process, factor out the handle-to-vm logic for the teardown path and reuse it for both the 'start' and 'finalise' stages of the teardown process.

---
 arch/arm64/kvm/hyp/nvhe/pkvm.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)

index 8b906217c4c3..3fd3b930beeb 100644
@@ -936,20 +936,27 @@ int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn)
 	return ret;
 }
 
+static struct pkvm_hyp_vm *get_pkvm_unref_hyp_vm_locked(pkvm_handle_t handle)
+{
+	struct pkvm_hyp_vm *hyp_vm;
+
+	hyp_assert_lock_held(&vm_table_lock);
+
+	hyp_vm = get_vm_by_handle(handle);
+	if (!hyp_vm || hyp_page_count(hyp_vm))
+		return NULL;
+
+	return hyp_vm;
+}
+
 int __pkvm_start_teardown_vm(pkvm_handle_t handle)
 {
 	struct pkvm_hyp_vm *hyp_vm;
 	int ret = 0;
 
 	hyp_spin_lock(&vm_table_lock);
-	hyp_vm = get_vm_by_handle(handle);
-	if (!hyp_vm) {
-		ret = -ENOENT;
-		goto unlock;
-	} else if (WARN_ON(hyp_page_count(hyp_vm))) {
-		ret = -EBUSY;
-		goto unlock;
-	} else if (hyp_vm->kvm.arch.pkvm.is_dying) {
+	hyp_vm = get_pkvm_unref_hyp_vm_locked(handle);
+	if (!hyp_vm || hyp_vm->kvm.arch.pkvm.is_dying) {
 		ret = -EINVAL;
 		goto unlock;
 	}

@@ -971,12 +978,9 @@ int __pkvm_finalize_teardown_vm(pkvm_handle_t handle)
 	int err;
 
 	hyp_spin_lock(&vm_table_lock);
-	hyp_vm = get_vm_by_handle(handle);
-	if (!hyp_vm) {
-		err = -ENOENT;
-		goto err_unlock;
-	} else if (!hyp_vm->kvm.arch.pkvm.is_dying) {
-		err = -EBUSY;
+	hyp_vm = get_pkvm_unref_hyp_vm_locked(handle);
+	if (!hyp_vm || !hyp_vm->kvm.arch.pkvm.is_dying) {
+		err = -EINVAL;
 		goto err_unlock;
 	}
 

2.53.0.1118.gaef5881109-goog

Now that completion of the teardown path requires a refcount of zero for the target VM, we can allow get_pkvm_hyp_vm() to take a reference on a dying VM, which is necessary to unshare pages with a non-protected VM during the teardown process itself.

Note that vCPUs belonging to a dying VM cannot be loaded and pages can only be reclaimed from a protected VM (via __pkvm_reclaim_dying_guest_page()) if the target VM is in the dying state.

---
 arch/arm64/kvm/hyp/nvhe/pkvm.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

index 3fd3b930beeb..b955da0e50bc 100644
@@ -309,14 +309,8 @@ struct pkvm_hyp_vm *get_pkvm_hyp_vm(pkvm_handle_t handle)
 
 	hyp_spin_lock(&vm_table_lock);
 	hyp_vm = get_vm_by_handle(handle);
-	if (!hyp_vm)
-		goto unlock;
-
-	if (hyp_vm->kvm.arch.pkvm.is_dying)
-		hyp_vm = NULL;
-	else
+	if (hyp_vm)
 		hyp_page_ref_inc(hyp_virt_to_page(hyp_vm));
-unlock:
 	hyp_spin_unlock(&vm_table_lock);
 
 	return hyp_vm;

2.53.0.1118.gaef5881109-goog

Now that the teardown of a VM cannot be finalised as long as a reference is held on the VM, rework __pkvm_reclaim_dying_guest_page() to hold a reference to the dying VM rather than take the global 'vm_table_lock' during the reclaim operation.

---
 arch/arm64/kvm/hyp/nvhe/pkvm.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

index b955da0e50bc..7ed96d64d611 100644
@@ -918,15 +918,16 @@ teardown_donated_memory(struct kvm_hyp_memcache *mc, void *addr, size_t size)
 
 int __pkvm_reclaim_dying_guest_page(pkvm_handle_t handle, u64 gfn)
 {
-	struct pkvm_hyp_vm *hyp_vm;
+	struct pkvm_hyp_vm *hyp_vm = get_pkvm_hyp_vm(handle);
 	int ret = -EINVAL;
 
-	hyp_spin_lock(&vm_table_lock);
-	hyp_vm = get_vm_by_handle(handle);
-	if (hyp_vm && hyp_vm->kvm.arch.pkvm.is_dying)
-		ret = __pkvm_host_reclaim_page_guest(gfn, hyp_vm);
-	hyp_spin_unlock(&vm_table_lock);
+	if (!hyp_vm)
+		return ret;
 
+	if (hyp_vm->kvm.arch.pkvm.is_dying)
+		ret = __pkvm_host_reclaim_page_guest(gfn, hyp_vm);
+
+	put_pkvm_hyp_vm(hyp_vm);
 	return ret;
 }
 

2.53.0.1118.gaef5881109-goog

This series fixes the warnings I reported with pKVM on N1DSP yesterday:

Applied to next, thanks!

[1/3] KVM: arm64: Prevent teardown finalisation of referenced 'hyp_vm'

      commit: 760299a1d8102b36bed5c25c5a3f94b5a0eee081
[2/3] KVM: arm64: Allow get_pkvm_hyp_vm() to take a reference to a dying VM
      commit: 2400696883870ec3fb0fb9925426c62a3383ca36
[3/3] KVM: arm64: Don't hold 'vm_table_lock' across guest page reclaim
      commit: bc20692f528b2ac8226bafe5b1db9a1f8be96dbf

Cheers,

	M.

Without deviation from the norm, progress is not possible.