Skip to contentSkip to conversation

KVM: s390: pv: implement lazy destroy for reboot

PATCH Superseded Claudio Imbrenda 3 participants 36 messages Jun 3, 2022 – Jun 20, 2022

Subsystems (5)
Documentation KVM s390 s390 MM +1 more subsystems

v11 of 16 Superseded Latest is v16 →

lore
Share & tools
lore

Claudio Imbrenda

author maintainer
#0 lore raw

v11 cover KVM: s390: pv: implement lazy destroy for reboot

Previously, when a protected VM was rebooted or when it was shut down, its memory was made unprotected, and then the protected VM itself was destroyed. Looping over the whole address space can take some time, considering the overhead of the various Ultravisor Calls (UVCs). This means that a reboot or a shutdown would take a potentially long amount of time, depending on the amount of used memory.

This patchseries implements a deferred destroy mechanism for protected guests. When a protected guest is destroyed, its memory can be cleared in background, allowing the guest to restart or terminate significantly faster than before.

There are 2 possibilities when a protected VM is torn down:
* it still has an address space associated (reboot case)
* it does not have an address space anymore (shutdown case)

For the reboot case, two new commands are available for the KVM_S390_PV_COMMAND:

KVM_PV_ASYNC_DISABLE_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. In this case the userspace process should issue a normal KVM_PV_DISABLE

KVM_PV_ASYNC_DISABLE: tears down the protected VM previously set aside for asychronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.

The idea is that userspace should first issue the KVM_PV_ASYNC_DISABLE_PREPARE command, and in case of success, create a new thread and issue KVM_PV_ASYNC_DISABLE from there. This also allows for proper accounting of the CPU time needed for the asynchronous teardown.

This means that the same address space can have memory belonging to more than one protected guest, although only one will be running, the others will in fact not even have any CPUs.

The shutdown case should be dealt with in userspace (e.g. using clone(CLONE_VM)).

A module parameter is also provided to disable the new functionality, which is otherwise enabled by default. This should not be an issue since the new functionality is opt-in anyway. This is mainly thought to aid debugging.

v10->v11
* rebase
* improve comments and patch descriptions
* rename s390_remove_old_asce to s390_unlist_old_asce
* rename DESTROY_LOOP_THRESHOLD to GATHER_GET_PAGES
* rename module parameter lazy_destroy to async_destroy
* move the WRITE_ONCE to be right after the UVC in patch 13
* improve handling leftover secure VMs in patch 14
* lock only when needed in patch 15, instead of always locking and then unlocking and locking again
* refactor should_export_before_import to make it more readable

v9->v10
* improved and expanded comments, fix typos
* add new patch: perform destroy configuration UVC before clearing memory for unconditional deinit_vm (instead of afterwards)
* explicitly initialize kvm->arch.pv.async_deinit in kvm_arch_init_vm
* do not try to call the destroy fast UVC in the MMU notifier if it is not available

v8->v9
* rebased
* added dependency on MMU_NOTIFIER for KVM in arch/s390/kvm/Kconfig
* add support for the Destroy Secure Configuration Fast UVC
* minor fixes

v7->v8
* switched patches 8 and 9
* improved comments, documentation and patch descriptions
* remove mm notifier when the struct kvm is torn down
* removed useless locks in the mm notifier
* use _ASCE_ORIGIN instead of PAGE_MASK for ASCEs
* cleanup of some compiler warnings
* remove some harmless but useless duplicate code
* the last parameter of __s390_uv_destroy_range is now bool
* rename the KVM capability to KVM_CAP_S390_PROTECTED_ASYNC_DISABLE

v6->v7
* moved INIT_LIST_HEAD inside spinlock in patch 1
* improved commit messages in patch 2
* added missing locks in patch 3
* added and expanded some comments in patch 11
* rebased

v5->v6
* completely reworked the series
* removed kernel thread for asynchronous teardown
* added new commands to KVM_S390_PV_COMMAND ioctl

v4->v5
* fixed and improved some patch descriptions
* added some comments to better explain what's going on
* use vma_lookup instead of find_vma
* rename is_protected to protected_count since now it's used as a counter

v3->v4
* added patch 2
* split patch 3
* removed the shutdown part -- will be a separate patchseries
* moved the patch introducing the module parameter

v2->v3
* added definitions for CC return codes for the UVC instruction
* improved make_secure_pte:
- renamed rc to cc
- added comments to explain why returning -EAGAIN is ok
* fixed kvm_s390_pv_replace_asce and kvm_s390_pv_remove_old_asce:
- renamed
- added locking
- moved to gmap.c
* do proper error management in do_secure_storage_access instead of trying again hoping to get a different exception
* fix outdated patch descriptions

v1->v2
* rebased on a more recent kernel
* improved/expanded some patch descriptions
* improves/expanded some comments
* added patch 1, which prevents stall notification when the system is under heavy load.
* rename some members of struct deferred_priv to improve readability
* avoid an use-after-free bug of the struct mm in case of shutdown
* add missing return when lazy destroy is disabled
* add support for OOM notifier

Claudio Imbrenda (19):
KVM: s390: pv: leak the topmost page table when destroy fails
KVM: s390: pv: handle secure storage violations for protected guests
KVM: s390: pv: handle secure storage exceptions for normal guests
KVM: s390: pv: refactor s390_reset_acc
KVM: s390: pv: usage counter instead of flag
KVM: s390: pv: add export before import
KVM: s390: pv: module parameter to fence asynchronous destroy
KVM: s390: pv: clear the state without memset
KVM: s390: pv: Add kvm_s390_cpus_from_pv to kvm-s390.h and add
documentation
KVM: s390: pv: add mmu_notifier
s390/mm: KVM: pv: when tearing down, try to destroy protected pages
KVM: s390: pv: refactoring of kvm_s390_pv_deinit_vm
KVM: s390: pv: destroy the configuration before its memory
KVM: s390: pv: cleanup leftover protected VMs if needed
KVM: s390: pv: asynchronous destroy for reboot
KVM: s390: pv: api documentation for asynchronous destroy
KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
KVM: s390: pv: avoid export before import if possible
KVM: s390: pv: support for Destroy fast UVC

 Documentation/virt/kvm/api.rst      |  25 ++-
 arch/s390/include/asm/gmap.h        |  39 +++-
 arch/s390/include/asm/kvm_host.h    |   4 +
 arch/s390/include/asm/mmu.h         |   2 +-
 arch/s390/include/asm/mmu_context.h |   2 +-
 arch/s390/include/asm/pgtable.h     |  21 +-
 arch/s390/include/asm/uv.h          |  11 +
 arch/s390/kernel/uv.c               |  79 +++++++
 arch/s390/kvm/Kconfig               |   1 +
 arch/s390/kvm/kvm-s390.c            |  80 ++++++-
 arch/s390/kvm/kvm-s390.h            |   3 +
 arch/s390/kvm/pv.c                  | 325 +++++++++++++++++++++++++++-
 arch/s390/mm/fault.c                |  23 +-
 arch/s390/mm/gmap.c                 | 173 ++++++++++++---
 include/uapi/linux/kvm.h            |   3 +
 15 files changed, 743 insertions(+), 48 deletions(-)
Signature · 1 line
2.36.1
author maintainer
#1 lore raw

PATCH v11 1/19 KVM: s390: pv: leak the topmost page table when destroy fails

Each secure guest must have a unique ASCE (address space control element); we must avoid that new guests use the same page for their ASCE, to avoid errors.

Since the ASCE mostly consists of the address of the topmost page table (plus some flags), we must not return that memory to the pool unless the ASCE is no longer in use.

Only a successful Destroy Secure Configuration UVC will make the ASCE reusable again.

If the Destroy Configuration UVC fails, the ASCE cannot be reused for a secure guest (either for the ASCE or for other memory areas). To avoid a collision, it must not be used again. This is a permanent error and the page becomes in practice unusable, so we set it aside and leak it. On failure we already leak other memory that belongs to the ultravisor (i.e. the variable and base storage for a guest) and not leaking the topmost page table was an oversight.

This error (and thus the leakage) should not happen unless the hardware is broken or KVM has some unknown serious bug.

Fixes 29b40f105ec8d55 ("KVM: s390: protvirt: Add initial vm and cpu lifecycle handling")
---
 arch/s390/include/asm/gmap.h |  2 +
 arch/s390/kvm/pv.c           |  9 ++--
 arch/s390/mm/gmap.c          | 86 ++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+), 3 deletions(-)
arch/s390/include/asm/gmap.h +2 -0
index 40264f60b0da..f4073106e1f3 100644
@@ -148,4 +148,6 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
 			     unsigned long gaddr, unsigned long vmaddr);
 int gmap_mark_unmergeable(void);
 void s390_reset_acc(struct mm_struct *mm);
+void s390_unlist_old_asce(struct gmap *gmap);
+int s390_replace_asce(struct gmap *gmap);
 #endif /* _ASM_S390_GMAP_H */
arch/s390/kvm/pv.c +6 -3
index cc7c9599f43e..8eee3fc414e5 100644
@@ -161,10 +161,13 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	atomic_set(&kvm->mm->context.is_protected, 0);
 	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x", *rc, *rrc);
 	WARN_ONCE(cc, "protvirt destroy vm failed rc %x rrc %x", *rc, *rrc);
-	/* Inteded memory leak on "impossible" error */
-	if (!cc)
+	/* Intended memory leak on "impossible" error */
+	if (!cc) {
 		kvm_s390_pv_dealloc_vm(kvm);
-	return cc ? -EIO : 0;
+		return 0;
+	}
+	s390_replace_asce(kvm->arch.gmap);
+	return -EIO;
 }
 
 int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
arch/s390/mm/gmap.c +86 -0

89 lines not on this page · raw message

Signature · 1 line
2.36.1
author maintainer
#2 lore raw

PATCH v11 2/19 KVM: s390: pv: handle secure storage violations for protected guests

In reply to Claudio Imbrenda

A secure storage violation is triggered when a protected guest tries to access secure memory that has been mapped erroneously, or that belongs to a different protected guest or to the ultravisor.

With upcoming patches, protected guests will be able to trigger secure storage violations in normal operation. This happens for example if a protected guest is rebooted with deferred destroy enabled and the new guest is also protected.

When the new protected guest touches pages that have not yet been destroyed, and thus are accounted to the previous protected guest, a secure storage violation is raised.

This patch adds handling of secure storage violations for protected guests.

This exception is handled by first trying to destroy the page, because it is expected to belong to a defunct protected guest where a destroy should be possible. Note that a secure page can only be destroyed if its protected VM does not have any CPUs, which only happens when the protected VM is being terminated. If that fails, a normal export of the page is attempted.

This means that pages that trigger the exception will be made non-secure (in one way or another) before attempting to use them again for a different secure guest.

---
 arch/s390/include/asm/uv.h |  1 +
 arch/s390/kernel/uv.c      | 55 ++++++++++++++++++++++++++++++++++++++
 arch/s390/mm/fault.c       | 10 +++++++
 3 files changed, 66 insertions(+)
arch/s390/include/asm/uv.h +1 -0
index cfea7b77a5b8..ba64e0be03bb 100644
@@ -378,6 +378,7 @@ static inline int is_prot_virt_host(void)
 }
 
 int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb);
+int gmap_destroy_page(struct gmap *gmap, unsigned long gaddr);
 int uv_destroy_owned_page(unsigned long paddr);
 int uv_convert_from_secure(unsigned long paddr);
 int uv_convert_owned_from_secure(unsigned long paddr);
arch/s390/kernel/uv.c +55 -0

61 lines not on this page · raw message

arch/s390/mm/fault.c +10 -0
index e173b6187ad5..af1ac49168fb 100644
@@ -837,6 +837,16 @@ NOKPROBE_SYMBOL(do_non_secure_storage_access);
 
 void do_secure_storage_violation(struct pt_regs *regs)
 {
+	unsigned long gaddr = regs->int_parm_long & __FAIL_ADDR_MASK;
+	struct gmap *gmap = (struct gmap *)S390_lowcore.gmap;
+
+	/*
+	 * If the VM has been rebooted, its address space might still contain
+	 * secure pages from the previous boot.
+	 * Clear the page so it can be reused.
+	 */
+	if (!gmap_destroy_page(gmap, gaddr))
+		return;
 	/*
 	 * Either KVM messed up the secure guest mapping or the same
 	 * page is mapped into multiple secure guests.
Signature · 1 line
2.36.1
author maintainer
#3 lore raw

PATCH v11 3/19 KVM: s390: pv: handle secure storage exceptions for normal guests

In reply to Claudio Imbrenda

With upcoming patches, normal guests might touch secure pages.

This patch extends the existing exception handler to convert the pages to non secure also when the exception is triggered by a normal guest.

This can happen for example when a secure guest reboots; the first stage of a secure guest is non secure, and in general a secure guest can reboot into non-secure mode.

If the secure memory of the previous boot has not been cleared up completely yet (which will be allowed to happen in an upcoming patch), a non-secure guest might touch secure memory, which will need to be handled properly.

This means that gmap faults must be handled and not cause termination of the process. The handling is the same as userspace accesses, it's enough to translate the gmap address to a user address and then let the normal user fault code handle it.

---
 arch/s390/mm/fault.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
arch/s390/mm/fault.c +12 -1
index af1ac49168fb..ee7871f770fb 100644
@@ -754,6 +754,7 @@ void do_secure_storage_access(struct pt_regs *regs)
 	struct vm_area_struct *vma;
 	struct mm_struct *mm;
 	struct page *page;
+	struct gmap *gmap;
 	int rc;
 
 	/*
@@ -783,6 +784,17 @@ void do_secure_storage_access(struct pt_regs *regs)
 	}
 
 	switch (get_fault_type(regs)) {
+	case GMAP_FAULT:
+		mm = current->mm;
+		gmap = (struct gmap *)S390_lowcore.gmap;
+		mmap_read_lock(mm);
+		addr = __gmap_translate(gmap, addr);
+		mmap_read_unlock(mm);
+		if (IS_ERR_VALUE(addr)) {
+			do_fault_error(regs, VM_ACCESS_FLAGS, VM_FAULT_BADMAP);
+			break;
+		}
+		fallthrough;
 	case USER_FAULT:
 		mm = current->mm;
 		mmap_read_lock(mm);
@@ -811,7 +823,6 @@ void do_secure_storage_access(struct pt_regs *regs)
 		if (rc)
 			BUG();
 		break;
-	case GMAP_FAULT:
 	default:
 		do_fault_error(regs, VM_READ | VM_WRITE, VM_FAULT_BADMAP);
 		WARN_ON_ONCE(1);
Signature · 1 line
2.36.1
author maintainer
#4 lore raw

PATCH v11 4/19 KVM: s390: pv: refactor s390_reset_acc

In reply to Claudio Imbrenda

Refactor s390_reset_acc so that it can be reused in upcoming patches.

We don't want to hold all the locks used in a walk_page_range for too long, and the destroy page UVC does take some time to complete. Therefore we quickly gather the pages to destroy, and then destroy them without holding all the locks.

The new refactored function optionally allows to return early without completing if a fatal signal is pending (and return and appropriate error code). Two wrappers are provided to call the new function.

---
 arch/s390/include/asm/gmap.h | 37 +++++++++++++-
 arch/s390/kvm/pv.c           | 12 ++++-
 arch/s390/mm/gmap.c          | 95 +++++++++++++++++++++++++-----------
 3 files changed, 112 insertions(+), 32 deletions(-)
arch/s390/include/asm/gmap.h +36 -1
index f4073106e1f3..21ae3b3c3bb5 100644
@@ -147,7 +147,42 @@ int gmap_mprotect_notify(struct gmap *, unsigned long start,
 void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long dirty_bitmap[4],
 			     unsigned long gaddr, unsigned long vmaddr);
 int gmap_mark_unmergeable(void);
-void s390_reset_acc(struct mm_struct *mm);
 void s390_unlist_old_asce(struct gmap *gmap);
 int s390_replace_asce(struct gmap *gmap);
+void s390_uv_destroy_pfns(unsigned long count, unsigned long *pfns);
+int __s390_uv_destroy_range(struct mm_struct *mm, unsigned long start,
+			    unsigned long end, bool interruptible);
+
+/**
+ * s390_uv_destroy_range - Destroy a range of pages in the given mm.
+ * @mm the mm on which to operate on
+ * @start the start of the range
+ * @end the end of the range
+ *
+ * This function will call cond_sched, so it should not generate stalls, but
+ * it will otherwise only return when it completed.
+ */
+static inline void s390_uv_destroy_range(struct mm_struct *mm, unsigned long start,
+					 unsigned long end)
+{
+	(void)__s390_uv_destroy_range(mm, start, end, false);
+}
+
+/**
+ * s390_uv_destroy_range_interruptible - Destroy a range of pages in the
+ * given mm, but stop when a fatal signal is received.
+ * @mm the mm on which to operate on
+ * @start the start of the range
+ * @end the end of the range
+ *
+ * This function will call cond_sched, so it should not generate stalls. If
+ * a fatal signal is received, it will return with -EINTR immediately,
+ * without finishing destroying the whole range. Upon successful
+ * completion, 0 is returned.
+ */
+static inline int s390_uv_destroy_range_interruptible(struct mm_struct *mm, unsigned long start,
+						      unsigned long end)
+{
+	return __s390_uv_destroy_range(mm, start, end, true);
+}
 #endif /* _ASM_S390_GMAP_H */
arch/s390/kvm/pv.c +10 -2
index 8eee3fc414e5..bcbe10862f9f 100644
@@ -12,6 +12,8 @@
 #include <asm/gmap.h>
 #include <asm/uv.h>
 #include <asm/mman.h>
+#include <linux/pagewalk.h>
+#include <linux/sched/mm.h>
 #include "kvm-s390.h"
 
 int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc)
@@ -152,8 +154,14 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 {
 	int cc;
 
-	/* make all pages accessible before destroying the guest */
-	s390_reset_acc(kvm->mm);
+	/*
+	 * if the mm still has a mapping, make all its pages accessible
+	 * before destroying the guest
+	 */
+	if (mmget_not_zero(kvm->mm)) {
+		s390_uv_destroy_range(kvm->mm, 0, TASK_SIZE);
+		mmput(kvm->mm);
+	}
 
 	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
 			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
arch/s390/mm/gmap.c +66 -29

110 lines not on this page · raw message

Signature · 1 line
2.36.1
author maintainer
#5 lore raw

PATCH v11 5/19 KVM: s390: pv: usage counter instead of flag

In reply to Claudio Imbrenda

Use the new protected_count field as a counter instead of the old is_protected flag. This will be used in upcoming patches.

Increment the counter when a secure configuration is created, and decrement it when it is destroyed. Previously the flag was set when the set secure parameters UVC was performed.

---
 arch/s390/include/asm/mmu.h         |  2 +-
 arch/s390/include/asm/mmu_context.h |  2 +-
 arch/s390/include/asm/pgtable.h     |  2 +-
 arch/s390/kvm/pv.c                  | 12 +++++++-----
 4 files changed, 10 insertions(+), 8 deletions(-)
arch/s390/include/asm/mmu.h +1 -1
index 82aae78e1315..1572b3634cdd 100644
@@ -18,7 +18,7 @@ typedef struct {
 	unsigned long asce_limit;
 	unsigned long vdso_base;
 	/* The mmu context belongs to a secure guest. */
-	atomic_t is_protected;
+	atomic_t protected_count;
 	/*
 	 * The following bitfields need a down_write on the mm
 	 * semaphore when they are written to. As they are only
arch/s390/include/asm/mmu_context.h +1 -1
index c7937f369e62..2a38af5a00c2 100644
@@ -26,7 +26,7 @@ static inline int init_new_context(struct task_struct *tsk,
 	INIT_LIST_HEAD(&mm->context.gmap_list);
 	cpumask_clear(&mm->context.cpu_attach_mask);
 	atomic_set(&mm->context.flush_count, 0);
-	atomic_set(&mm->context.is_protected, 0);
+	atomic_set(&mm->context.protected_count, 0);
 	mm->context.gmap_asce = 0;
 	mm->context.flush_mm = 0;
 #ifdef CONFIG_PGSTE
arch/s390/include/asm/pgtable.h +1 -1
index a397b072a580..f16403ba81ec 100644
@@ -525,7 +525,7 @@ static inline int mm_has_pgste(struct mm_struct *mm)
 static inline int mm_is_protected(struct mm_struct *mm)
 {
 #ifdef CONFIG_PGSTE
-	if (unlikely(atomic_read(&mm->context.is_protected)))
+	if (unlikely(atomic_read(&mm->context.protected_count)))
 		return 1;
 #endif
 	return 0;
arch/s390/kvm/pv.c +7 -5
index bcbe10862f9f..f3134d79f8e1 100644
@@ -166,7 +166,8 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
 			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
 	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
-	atomic_set(&kvm->mm->context.is_protected, 0);
+	if (!cc)
+		atomic_dec(&kvm->mm->context.protected_count);
 	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x", *rc, *rrc);
 	WARN_ONCE(cc, "protvirt destroy vm failed rc %x rrc %x", *rc, *rrc);
 	/* Intended memory leak on "impossible" error */
@@ -208,11 +209,14 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	/* Outputs */
 	kvm->arch.pv.handle = uvcb.guest_handle;
 
+	atomic_inc(&kvm->mm->context.protected_count);
 	if (cc) {
-		if (uvcb.header.rc & UVC_RC_NEED_DESTROY)
+		if (uvcb.header.rc & UVC_RC_NEED_DESTROY) {
 			kvm_s390_pv_deinit_vm(kvm, &dummy, &dummy);
-		else
+		} else {
+			atomic_dec(&kvm->mm->context.protected_count);
 			kvm_s390_pv_dealloc_vm(kvm);
+		}
 		return -EIO;
 	}
 	kvm->arch.gmap->guest_handle = uvcb.guest_handle;
@@ -235,8 +239,6 @@ int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
 	*rrc = uvcb.header.rrc;
 	KVM_UV_EVENT(kvm, 3, "PROTVIRT VM SET PARMS: rc %x rrc %x",
 		     *rc, *rrc);
-	if (!cc)
-		atomic_set(&kvm->mm->context.is_protected, 1);
 	return cc ? -EINVAL : 0;
 }
 
Signature · 1 line
2.36.1
author maintainer
#6 lore raw

PATCH v11 6/19 KVM: s390: pv: add export before import

In reply to Claudio Imbrenda

Due to upcoming changes, it will be possible to temporarily have multiple protected VMs in the same address space, although only one will be actually active.

In that scenario, it is necessary to perform an export of every page that is to be imported, since the hardware does not allow a page belonging to a protected guest to be imported into a different protected guest.

This also applies to pages that are shared, and thus accessible by the host.

---
 arch/s390/kernel/uv.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/s390/kernel/uv.c +22 -0
index 2754471cc789..02aca3c5dce1 100644
@@ -234,6 +234,26 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,
 	return uvcb->rc == 0x10a ? -ENXIO : -EINVAL;
 }
 
+/**
+ * should_export_before_import - Determine whether an export is needed
+ * before an import-like operation
+ * @uvcb the Ultravisor control block of the UVC to be performed
+ * @mm the mm of the process
+ *
+ * Although considered as one, the Unpin Page UVC is not an actual import,
+ * so it is not affected.
+ *
+ * No export is needed also when there is only one protected VM, because the
+ * page cannot belong to the wrong VM in that case (there is no "other VM"
+ * it can belong to).
+ */
+static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
+{
+	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
+		return false;
+	return atomic_read(&mm->context.protected_count) > 1;
+}
+
 /*
  * Requests the Ultravisor to make a page accessible to a guest.
  * If it's brought in the first time, it will be cleared. If
@@ -277,6 +297,8 @@ int gmap_make_secure(struct gmap *gmap, unsigned long gaddr, void *uvcb)
 
 	lock_page(page);
 	ptep = get_locked_pte(gmap->mm, uaddr, &ptelock);
+	if (should_export_before_import(uvcb, gmap->mm))
+		uv_convert_from_secure(page_to_phys(page));
 	rc = make_secure_pte(ptep, uaddr, page, uvcb);
 	pte_unmap_unlock(ptep, ptelock);
 	unlock_page(page);
Signature · 1 line
2.36.1
author maintainer
#7 lore raw

PATCH v11 7/19 KVM: s390: pv: module parameter to fence asynchronous destroy

In reply to Claudio Imbrenda

Add the module parameter "async_destroy", to allow the asynchronous destroy mechanism to be switched off. This might be useful for debugging purposes.

The parameter is enabled by default.

---
 arch/s390/kvm/kvm-s390.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/s390/kvm/kvm-s390.c +5 -0
index 76ad6408cb2c..49e27b5d7c3a 100644
@@ -206,6 +206,11 @@ unsigned int diag9c_forwarding_hz;
 module_param(diag9c_forwarding_hz, uint, 0644);
 MODULE_PARM_DESC(diag9c_forwarding_hz, "Maximum diag9c forwarding per second, 0 to turn off");
 
+/* allow asynchronous deinit for protected guests, enable by default */
+static int async_destroy = 1;
+module_param(async_destroy, int, 0444);
+MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests");
+
 /*
  * For now we handle at most 16 double words as this is what the s390 base
  * kernel handles and stores in the prefix page. If we ever need to go beyond
Signature · 1 line
2.36.1
maintainer
#8 lore raw

Re: [PATCH v11 07/19] KVM: s390: pv: module parameter to fence asynchronous destroy

Quoted 8 lines · Claudio Imbrenda

Add the module parameter "async_destroy", to allow the asynchronous destroy mechanism to be switched off. This might be useful for debugging purposes.

The parameter is enabled by default.

Normally this would be one of the last patches in the series, no?

Quoted 20 lines
---
  arch/s390/kvm/kvm-s390.c | 5 +++++
  1 file changed, 5 insertions(+)
arch/s390/kvm/kvm-s390.c +5 -0
index 76ad6408cb2c..49e27b5d7c3a 100644
@@ -206,6 +206,11 @@ unsigned int diag9c_forwarding_hz;
  module_param(diag9c_forwarding_hz, uint, 0644);
  MODULE_PARM_DESC(diag9c_forwarding_hz, "Maximum diag9c forwarding per second, 0 to turn off");
 
+/* allow asynchronous deinit for protected guests, enable by default */
+static int async_destroy = 1;
+module_param(async_destroy, int, 0444);
+MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests");
+
  /*
   * For now we handle at most 16 double words as this is what the s390 base
   * kernel handles and stores in the prefix page. If we ever need to go beyond
author maintainer
#9 lore raw

Re: [PATCH v11 07/19] KVM: s390: pv: module parameter to fence asynchronous destroy

Quoted 11 lines · On Wed, 15 Jun 2022 11:53:17 +0200 Janosch Frank <[email protected]> wrote:
Quoted 8 lines · Claudio Imbrenda

Add the module parameter "async_destroy", to allow the asynchronous destroy mechanism to be switched off. This might be useful for debugging purposes.

The parameter is enabled by default.

Normally this would be one of the last patches in the series, no?

I need the variable to be already defined, because the subsequent patches use it to fence things

Quoted 22 lines
Quoted 21 lines
---
  arch/s390/kvm/kvm-s390.c | 5 +++++
  1 file changed, 5 insertions(+)
arch/s390/kvm/kvm-s390.c +5 -0
index 76ad6408cb2c..49e27b5d7c3a 100644
@@ -206,6 +206,11 @@ unsigned int diag9c_forwarding_hz;
  module_param(diag9c_forwarding_hz, uint, 0644);
  MODULE_PARM_DESC(diag9c_forwarding_hz, "Maximum diag9c forwarding per second, 0 to turn off");
 
+/* allow asynchronous deinit for protected guests, enable by default */
+static int async_destroy = 1;
+module_param(async_destroy, int, 0444);
+MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests");
+
  /*
   * For now we handle at most 16 double words as this is what the s390 base
   * kernel handles and stores in the prefix page. If we ever need to go beyond  
author maintainer
#10 lore raw

PATCH v11 8/19 KVM: s390: pv: clear the state without memset

In reply to Claudio Imbrenda

Do not use memset to clean the whole struct kvm_s390_pv; instead, explicitly clear the fields that need to be cleared.

Upcoming patches will introduce new fields in the struct kvm_s390_pv that will not need to be cleared.

---
 arch/s390/kvm/pv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/s390/kvm/pv.c +9 -1
index f3134d79f8e1..9eca80afedce 100644
@@ -16,6 +16,14 @@
 #include <linux/sched/mm.h>
 #include "kvm-s390.h"
 
+static void kvm_s390_clear_pv_state(struct kvm *kvm)
+{
+	kvm->arch.pv.handle = 0;
+	kvm->arch.pv.guest_len = 0;
+	kvm->arch.pv.stor_base = 0;
+	kvm->arch.pv.stor_var = NULL;
+}
+
 int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc)
 {
 	int cc;
@@ -110,7 +118,7 @@ static void kvm_s390_pv_dealloc_vm(struct kvm *kvm)
 	vfree(kvm->arch.pv.stor_var);
 	free_pages(kvm->arch.pv.stor_base,
 		   get_order(uv_info.guest_base_stor_len));
-	memset(&kvm->arch.pv, 0, sizeof(kvm->arch.pv));
+	kvm_s390_clear_pv_state(kvm);
 }
 
 static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
Signature · 1 line
2.36.1
author maintainer
#11 lore raw

PATCH v11 9/19 KVM: s390: pv: Add kvm_s390_cpus_from_pv to kvm-s390.h and add documentation

In reply to Claudio Imbrenda

Future changes make it necessary to call this function from pv.c.

While we are at it, let's properly document kvm_s390_cpus_from_pv() and kvm_s390_cpus_to_pv().

---
 arch/s390/kvm/kvm-s390.c | 26 +++++++++++++++++++++++++-
 arch/s390/kvm/kvm-s390.h |  1 +
 2 files changed, 26 insertions(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +25 -1
index 49e27b5d7c3a..8973985593a9 100644
@@ -2175,7 +2175,20 @@ static int kvm_s390_set_cmma_bits(struct kvm *kvm,
 	return r;
 }
 
-static int kvm_s390_cpus_from_pv(struct kvm *kvm, u16 *rcp, u16 *rrcp)
+/**
+ * kvm_s390_cpus_from_pv - Convert all protected vCPUs in a protected VM to
+ * non protected.
+ * @kvm the VM whose protected vCPUs are to be converted
+ * @rcp return value for the RC field of the UVC (in case of error)
+ * @rrcp return value for the RRC field of the UVC (in case of error)
+ *
+ * Does not stop in case of error, tries to convert as many
+ * CPUs as possible. In case of error, the RC and RRC of the last error are
+ * returned.
+ *
+ * Return: 0 in case of success, otherwise -EIO
+ */
+int kvm_s390_cpus_from_pv(struct kvm *kvm, u16 *rcp, u16 *rrcp)
 {
 	struct kvm_vcpu *vcpu;
 	u16 rc, rrc;
@@ -2205,6 +2218,17 @@ static int kvm_s390_cpus_from_pv(struct kvm *kvm, u16 *rcp, u16 *rrcp)
 	return ret;
 }
 
+/**
+ * kvm_s390_cpus_to_pv - Convert all non-protected vCPUs in a protected VM
+ * to protected.
+ * @kvm the VM whose protected vCPUs are to be converted
+ * @rcp return value for the RC field of the UVC (in case of error)
+ * @rrcp return value for the RRC field of the UVC (in case of error)
+ *
+ * Tries to undo the conversion in case of error.
+ *
+ * Return: 0 in case of success, otherwise -EIO
+ */
 static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
 {
 	unsigned long i;
arch/s390/kvm/kvm-s390.h +1 -0
index 497d52a83c78..d3abedafa7a8 100644
@@ -374,6 +374,7 @@ int kvm_s390_vcpu_setup_cmma(struct kvm_vcpu *vcpu);
 void kvm_s390_vcpu_unsetup_cmma(struct kvm_vcpu *vcpu);
 void kvm_s390_set_cpu_timer(struct kvm_vcpu *vcpu, __u64 cputm);
 __u64 kvm_s390_get_cpu_timer(struct kvm_vcpu *vcpu);
+int kvm_s390_cpus_from_pv(struct kvm *kvm, u16 *rcp, u16 *rrcp);
 
 /* implemented in diag.c */
 int kvm_s390_handle_diag(struct kvm_vcpu *vcpu);
Signature · 1 line
2.36.1
author maintainer
#12 lore raw

PATCH v11 10/19 KVM: s390: pv: add mmu_notifier

In reply to Claudio Imbrenda

Add an mmu_notifier for protected VMs. The callback function is triggered when the mm is torn down, and will attempt to convert all protected vCPUs to non-protected. This allows the mm teardown to use the destroy page UVC instead of export.

Also make KVM select CONFIG_MMU_NOTIFIER, needed to use mmu_notifiers.

---
 arch/s390/include/asm/kvm_host.h |  2 ++
 arch/s390/kvm/Kconfig            |  1 +
 arch/s390/kvm/kvm-s390.c         | 10 ++++++++++
 arch/s390/kvm/pv.c               | 26 ++++++++++++++++++++++++++
 4 files changed, 39 insertions(+)
arch/s390/include/asm/kvm_host.h +2 -0
index 766028d54a3e..5824efe5fc9d 100644
@@ -19,6 +19,7 @@
 #include <linux/kvm.h>
 #include <linux/seqlock.h>
 #include <linux/module.h>
+#include <linux/mmu_notifier.h>
 #include <asm/debug.h>
 #include <asm/cpu.h>
 #include <asm/fpu/api.h>
@@ -923,6 +924,7 @@ struct kvm_s390_pv {
 	u64 guest_len;
 	unsigned long stor_base;
 	void *stor_var;
+	struct mmu_notifier mmu_notifier;
 };
 
 struct kvm_arch{
arch/s390/kvm/Kconfig +1 -0
index 2e84d3922f7c..33f4ff909476 100644
@@ -34,6 +34,7 @@ config KVM
 	select SRCU
 	select KVM_VFIO
 	select INTERVAL_TREE
+	select MMU_NOTIFIER
 	help
 	  Support hosting paravirtualized guest machines using the SIE
 	  virtualization capability on the mainframe. This should work
arch/s390/kvm/kvm-s390.c +10 -0
index 8973985593a9..fe1fa896def7 100644
@@ -31,6 +31,7 @@
 #include <linux/sched/signal.h>
 #include <linux/string.h>
 #include <linux/pgtable.h>
+#include <linux/mmu_notifier.h>
 
 #include <asm/asm-offsets.h>
 #include <asm/lowcore.h>
@@ -2937,6 +2938,15 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 	 */
 	if (kvm_s390_pv_get_handle(kvm))
 		kvm_s390_pv_deinit_vm(kvm, &rc, &rrc);
+	/*
+	 * Remove the mmu notifier only when the whole KVM VM is torn down,
+	 * and only if one was registered to begin with. If the VM is
+	 * currently not protected, but has been previously been protected,
+	 * then it's possible that the notifier is still registered.
+	 */
+	if (kvm->arch.pv.mmu_notifier.ops)
+		mmu_notifier_unregister(&kvm->arch.pv.mmu_notifier, kvm->mm);
+
 	debug_unregister(kvm->arch.dbf);
 	free_page((unsigned long)kvm->arch.sie_page2);
 	if (!kvm_is_ucontrol(kvm))
arch/s390/kvm/pv.c +26 -0
index 9eca80afedce..da1bae111fb1 100644
@@ -14,6 +14,7 @@
 #include <asm/mman.h>
 #include <linux/pagewalk.h>
 #include <linux/sched/mm.h>
+#include <linux/mmu_notifier.h>
 #include "kvm-s390.h"
 
 static void kvm_s390_clear_pv_state(struct kvm *kvm)
@@ -187,6 +188,26 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	return -EIO;
 }
 
+static void kvm_s390_pv_mmu_notifier_release(struct mmu_notifier *subscription,
+					     struct mm_struct *mm)
+{
+	struct kvm *kvm = container_of(subscription, struct kvm, arch.pv.mmu_notifier);
+	u16 dummy;
+
+	/*
+	 * No locking is needed since this is the last thread of the last user of this
+	 * struct mm.
+	 * When the struct kvm gets deinitialized, this notifier is also
+	 * unregistered. This means that if this notifier runs, then the
+	 * struct kvm is still valid.
+	 */
+	kvm_s390_cpus_from_pv(kvm, &dummy, &dummy);
+}
+
+static const struct mmu_notifier_ops kvm_s390_pv_mmu_notifier_ops = {
+	.release = kvm_s390_pv_mmu_notifier_release,
+};
+
 int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 {
 	struct uv_cb_cgc uvcb = {
@@ -228,6 +249,11 @@ int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 		return -EIO;
 	}
 	kvm->arch.gmap->guest_handle = uvcb.guest_handle;
+	/* Add the notifier only once. No races because we hold kvm->lock */
+	if (kvm->arch.pv.mmu_notifier.ops != &kvm_s390_pv_mmu_notifier_ops) {
+		kvm->arch.pv.mmu_notifier.ops = &kvm_s390_pv_mmu_notifier_ops;
+		mmu_notifier_register(&kvm->arch.pv.mmu_notifier, kvm->mm);
+	}
 	return 0;
 }
 
Signature · 1 line
2.36.1
Reviewed-by on v11 10/19
#13 lore raw

Re: [PATCH v11 10/19] KVM: s390: pv: add mmu_notifier

Quoted 9 lines · On Fri, 3 Jun 2022 08:56:36 +0200 Claudio Imbrenda <[email protected]>…

Add an mmu_notifier for protected VMs. The callback function is triggered when the mm is torn down, and will attempt to convert all protected vCPUs to non-protected. This allows the mm teardown to use the destroy page UVC instead of export.

Also make KVM select CONFIG_MMU_NOTIFIER, needed to use mmu_notifiers.

author maintainer
#14 lore raw

PATCH v11 11/19 s390/mm: KVM: pv: when tearing down, try to destroy protected pages

In reply to Claudio Imbrenda

When ptep_get_and_clear_full is called for a mm teardown, we will now attempt to destroy the secure pages. This will be faster than export.

In case it was not a teardown, or if for some reason the destroy page UVC failed, we try with an export page, like before.

---
 arch/s390/include/asm/pgtable.h | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
arch/s390/include/asm/pgtable.h +16 -3
index f16403ba81ec..cf81acf3879c 100644
@@ -1182,9 +1182,22 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,
 	} else {
 		res = ptep_xchg_lazy(mm, addr, ptep, __pte(_PAGE_INVALID));
 	}
-	/* At this point the reference through the mapping is still present */
-	if (mm_is_protected(mm) && pte_present(res))
-		uv_convert_owned_from_secure(pte_val(res) & PAGE_MASK);
+	/* Nothing to do */
+	if (!mm_is_protected(mm) || !pte_present(res))
+		return res;
+	/*
+	 * At this point the reference through the mapping is still present.
+	 * The notifier should have destroyed all protected vCPUs at this
+	 * point, so the destroy should be successful.
+	 */
+	if (full && !uv_destroy_owned_page(pte_val(res) & PAGE_MASK))
+		return res;
+	/*
+	 * If something went wrong and the page could not be destroyed, or
+	 * if this is not a mm teardown, the slower export is used as
+	 * fallback instead.
+	 */
+	uv_convert_owned_from_secure(pte_val(res) & PAGE_MASK);
 	return res;
 }
 
Signature · 1 line
2.36.1
Reviewed-by on v11 11/19
#15 lore raw

Re: [PATCH v11 11/19] s390/mm: KVM: pv: when tearing down, try to destroy protected pages

Quoted 8 lines · On Fri, 3 Jun 2022 08:56:37 +0200 Claudio Imbrenda <[email protected]>…

When ptep_get_and_clear_full is called for a mm teardown, we will now attempt to destroy the secure pages. This will be faster than export.

In case it was not a teardown, or if for some reason the destroy page UVC failed, we try with an export page, like before.

author maintainer
#16 lore raw

PATCH v11 12/19 KVM: s390: pv: refactoring of kvm_s390_pv_deinit_vm

In reply to Claudio Imbrenda

Refactor kvm_s390_pv_deinit_vm to improve readability and simplify the improvements that are coming in subsequent patches.

No functional change intended.

[note: this can potentially be squashed into the next patch, I factored it out to simplify the review process]

---
 arch/s390/kvm/pv.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/s390/kvm/pv.c +8 -8
index da1bae111fb1..a389555d62e7 100644
@@ -175,17 +175,17 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
 			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
 	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
-	if (!cc)
-		atomic_dec(&kvm->mm->context.protected_count);
-	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x", *rc, *rrc);
-	WARN_ONCE(cc, "protvirt destroy vm failed rc %x rrc %x", *rc, *rrc);
-	/* Intended memory leak on "impossible" error */
 	if (!cc) {
+		atomic_dec(&kvm->mm->context.protected_count);
 		kvm_s390_pv_dealloc_vm(kvm);
-		return 0;
+	} else {
+		/* Intended memory leak on "impossible" error */
+		s390_replace_asce(kvm->arch.gmap);
 	}
-	s390_replace_asce(kvm->arch.gmap);
-	return -EIO;
+	KVM_UV_EVENT(kvm, 3, "PROTVIRT DESTROY VM: rc %x rrc %x", *rc, *rrc);
+	WARN_ONCE(cc, "protvirt destroy vm failed rc %x rrc %x", *rc, *rrc);
+
+	return cc ? -EIO : 0;
 }
 
 static void kvm_s390_pv_mmu_notifier_release(struct mmu_notifier *subscription,
Signature · 1 line
2.36.1
author maintainer
#17 lore raw

PATCH v11 13/19 KVM: s390: pv: destroy the configuration before its memory

In reply to Claudio Imbrenda

Move the Destroy Secure Configuration UVC before the loop to destroy the memory. If the protected VM has memory, it will be cleaned up and made accessible by the Destroy Secure Configuraion UVC. The struct page for the relevant pages will still have the protected bit set, so the loop is still needed to clean that up.

Switching the order of those two operations does not change the outcome, but it is significantly faster.

---
 arch/s390/kvm/pv.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/s390/kvm/pv.c +3 -3
index a389555d62e7..6cffea26c47f 100644
@@ -163,6 +163,9 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 {
 	int cc;
 
+	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
+			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
+	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
 	/*
 	 * if the mm still has a mapping, make all its pages accessible
 	 * before destroying the guest
@@ -172,9 +175,6 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
 		mmput(kvm->mm);
 	}
 
-	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
-			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
-	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
 	if (!cc) {
 		atomic_dec(&kvm->mm->context.protected_count);
 		kvm_s390_pv_dealloc_vm(kvm);
Signature · 1 line
2.36.1
Reviewed-by on v11 13/19
#18 lore raw

Re: [PATCH v11 13/19] KVM: s390: pv: destroy the configuration before its memory

Quoted 10 lines · On Fri, 3 Jun 2022 08:56:39 +0200 Claudio Imbrenda <[email protected]>…

Move the Destroy Secure Configuration UVC before the loop to destroy the memory. If the protected VM has memory, it will be cleaned up and made accessible by the Destroy Secure Configuraion UVC. The struct page for the relevant pages will still have the protected bit set, so the loop is still needed to clean that up.

Switching the order of those two operations does not change the outcome, but it is significantly faster.

maintainer Reviewed-by on v11 13/19
#19 lore raw

Re: [PATCH v11 13/19] KVM: s390: pv: destroy the configuration before its memory

In reply to Claudio Imbrenda
Quoted 3 lines · Claudio Imbrenda

Move the Destroy Secure Configuration UVC before the loop to destroy the memory. If the protected VM has memory, it will be cleaned up and made accessible by the Destroy Secure Configuraion UVC. The struct

s/Configuraion/Configuration/

Quoted 7 lines

page for the relevant pages will still have the protected bit set, so the loop is still needed to clean that up.

Switching the order of those two operations does not change the outcome, but it is significantly faster.

Quoted 28 lines
---
  arch/s390/kvm/pv.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
arch/s390/kvm/pv.c +3 -3
index a389555d62e7..6cffea26c47f 100644
@@ -163,6 +163,9 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
  {
  	int cc;
 
+	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
+			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
+	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
  	/*
  	 * if the mm still has a mapping, make all its pages accessible
  	 * before destroying the guest
@@ -172,9 +175,6 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
  		mmput(kvm->mm);
  	}
 
-	cc = uv_cmd_nodata(kvm_s390_pv_get_handle(kvm),
-			   UVC_CMD_DESTROY_SEC_CONF, rc, rrc);
-	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
  	if (!cc) {
  		atomic_dec(&kvm->mm->context.protected_count);
  		kvm_s390_pv_dealloc_vm(kvm);
author maintainer
#20 lore raw

PATCH v11 14/19 KVM: s390: pv: cleanup leftover protected VMs if needed

In reply to Claudio Imbrenda

In upcoming patches it will be possible to start tearing down a protected VM, and finish the teardown concurrently in a different thread.

Protected VMs that are pending for tear down ("leftover") need to be cleaned properly when the userspace process (e.g. qemu) terminates.

This patch makes sure that all "leftover" protected VMs are always properly torn down.

---
 arch/s390/include/asm/kvm_host.h |   2 +
 arch/s390/kvm/kvm-s390.c         |   2 +
 arch/s390/kvm/pv.c               | 109 ++++++++++++++++++++++++++++---
 3 files changed, 104 insertions(+), 9 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index 5824efe5fc9d..cca8e05e0a71 100644
@@ -924,6 +924,8 @@ struct kvm_s390_pv {
 	u64 guest_len;
 	unsigned long stor_base;
 	void *stor_var;
+	void *prepared_for_async_deinit;
+	struct list_head need_cleanup;
 	struct mmu_notifier mmu_notifier;
 };
 
arch/s390/kvm/kvm-s390.c +2 -0
index fe1fa896def7..369de8377116 100644
@@ -2890,6 +2890,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	kvm_s390_vsie_init(kvm);
 	if (use_gisa)
 		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.prepared_for_async_deinit = NULL;
 	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
 	return 0;
arch/s390/kvm/pv.c +100 -9

135 lines not on this page · raw message

Signature · 1 line
2.36.1
maintainer
#21 lore raw

Re: [PATCH v11 14/19] KVM: s390: pv: cleanup leftover protected VMs if needed

Quoted 3 lines · Claudio Imbrenda

In upcoming patches it will be possible to start tearing down a protected VM, and finish the teardown concurrently in a different thread.

s/,/
s/the/its/

Quoted 6 lines

Protected VMs that are pending for tear down ("leftover") need to be cleaned properly when the userspace process (e.g. qemu) terminates.

This patch makes sure that all "leftover" protected VMs are always properly torn down.

So we're handling the kvm_arch_destroy_vm() case here, right? Maybe add that in a more prominent way and rework the subject:

KVM: s390: pv: cleanup leftover PV VM shells on VM shutdown

Quoted 39 lines
---
  arch/s390/include/asm/kvm_host.h |   2 +
  arch/s390/kvm/kvm-s390.c         |   2 +
  arch/s390/kvm/pv.c               | 109 ++++++++++++++++++++++++++++---
  3 files changed, 104 insertions(+), 9 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index 5824efe5fc9d..cca8e05e0a71 100644
@@ -924,6 +924,8 @@ struct kvm_s390_pv {
  	u64 guest_len;
  	unsigned long stor_base;
  	void *stor_var;
+	void *prepared_for_async_deinit;
+	struct list_head need_cleanup;
  	struct mmu_notifier mmu_notifier;
  };
 
arch/s390/kvm/kvm-s390.c +2 -0
index fe1fa896def7..369de8377116 100644
@@ -2890,6 +2890,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  	kvm_s390_vsie_init(kvm);
  	if (use_gisa)
  		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.prepared_for_async_deinit = NULL;
  	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
  	return 0;
arch/s390/kvm/pv.c +0 -0
index 6cffea26c47f..8471c17d538c 100644

arch/s390/kvm/pv.c line 21

Quoted 5 lines
arch/s390/kvm/pv.c +2 -0
@@ -17,6 +17,19 @@
  #include <linux/mmu_notifier.h>
  #include "kvm-s390.h"
 
+/**
+ * @struct leftover_pv_vm

Any other ideas on naming these VMs?
Also I'd turn that around: pv_vm_leftover

arch/s390/kvm/pv.c line 32

Quoted 11 lines
arch/s390/kvm/pv.c +11 -0
+ * Represents a "leftover" protected VM that is still registered with the
+ * Ultravisor, but which does not correspond any longer to an active KVM VM.
+ */
+struct leftover_pv_vm {
+	struct list_head list;
+	unsigned long old_gmap_table;
+	u64 handle;
+	void *stor_var;
+	unsigned long stor_base;
+};
+

I think we should switch this patch and the next one and add this struct to the next patch. The list work below makes more sense once the next patch has been read.

Quoted 9 lines
arch/s390/kvm/pv.c +0 -0
  static void kvm_s390_clear_pv_state(struct kvm *kvm)
  {
  	kvm->arch.pv.handle = 0;
@@ -158,23 +171,88 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
  	return -ENOMEM;
  }
 
 
 
author maintainer
#22 lore raw

Re: [PATCH v11 14/19] KVM: s390: pv: cleanup leftover protected VMs if needed

Quoted 7 lines · On Wed, 15 Jun 2022 11:59:36 +0200 Janosch Frank <[email protected]> wrote:
Quoted 3 lines · Claudio Imbrenda

In upcoming patches it will be possible to start tearing down a protected VM, and finish the teardown concurrently in a different thread.

s/,/
s/the/its/

will fix

Quoted 9 lines
Quoted 7 lines

Protected VMs that are pending for tear down ("leftover") need to be cleaned properly when the userspace process (e.g. qemu) terminates.

This patch makes sure that all "leftover" protected VMs are always properly torn down.

So we're handling the kvm_arch_destroy_vm() case here, right?

yes

Quoted 3 lines

Maybe add that in a more prominent way and rework the subject:

KVM: s390: pv: cleanup leftover PV VM shells on VM shutdown

ok, I'll change the description and rework the subject

Quoted 43 lines
Quoted 43 lines
---
  arch/s390/include/asm/kvm_host.h |   2 +
  arch/s390/kvm/kvm-s390.c         |   2 +
  arch/s390/kvm/pv.c               | 109 ++++++++++++++++++++++++++++---
  3 files changed, 104 insertions(+), 9 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index 5824efe5fc9d..cca8e05e0a71 100644
@@ -924,6 +924,8 @@ struct kvm_s390_pv {
  	u64 guest_len;
  	unsigned long stor_base;
  	void *stor_var;
+	void *prepared_for_async_deinit;
+	struct list_head need_cleanup;
  	struct mmu_notifier mmu_notifier;
  };
 
arch/s390/kvm/kvm-s390.c +2 -0
index fe1fa896def7..369de8377116 100644
@@ -2890,6 +2890,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  	kvm_s390_vsie_init(kvm);
  	if (use_gisa)
  		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.prepared_for_async_deinit = NULL;
  	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
  	return 0;
arch/s390/kvm/pv.c +0 -0
index 6cffea26c47f..8471c17d538c 100644
@@ -17,6 +17,19 @@
  #include <linux/mmu_notifier.h>
  #include "kvm-s390.h"
 
Quoted 4 lines

arch/s390/kvm/pv.c line 21

Quoted 2 lines
arch/s390/kvm/pv.c +2 -0
+/**
+ * @struct leftover_pv_vm  

Any other ideas on naming these VMs?

not really

Quoted 1 line

Also I'd turn that around: pv_vm_leftover

I mean, it's a leftover protected VM, it felt more natural to name it that way

Quoted 16 lines

arch/s390/kvm/pv.c line 32

Quoted 12 lines
arch/s390/kvm/pv.c +11 -0
+ * Represents a "leftover" protected VM that is still registered with the
+ * Ultravisor, but which does not correspond any longer to an active KVM VM.
+ */
+struct leftover_pv_vm {
+	struct list_head list;
+	unsigned long old_gmap_table;
+	u64 handle;
+	void *stor_var;
+	unsigned long stor_base;
+};
+  

I think we should switch this patch and the next one and add this struct to the next patch. The list work below makes more sense once the next patch has been read.

but the next patch will leave leftovers in some circumstances, and those won't be cleaned up without this patch.

having this patch first means that when the next patch is applied, the leftovers are already taken care of

Quoted 10 lines
Quoted 9 lines
arch/s390/kvm/pv.c +0 -0
  static void kvm_s390_clear_pv_state(struct kvm *kvm)
  {
  	kvm->arch.pv.handle = 0;
@@ -158,23 +171,88 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
  	return -ENOMEM;
  }
 
 
 
maintainer
#23 lore raw

Re: [PATCH v11 14/19] KVM: s390: pv: cleanup leftover protected VMs if needed

Quoted 103 lines · Claudio Imbrenda
Quoted 7 lines · On Wed, 15 Jun 2022 11:59:36 +0200 Janosch Frank <[email protected]> wrote:
Quoted 3 lines · Claudio Imbrenda

In upcoming patches it will be possible to start tearing down a protected VM, and finish the teardown concurrently in a different thread.

s/,/
s/the/its/

will fix

Quoted 9 lines
Quoted 7 lines

Protected VMs that are pending for tear down ("leftover") need to be cleaned properly when the userspace process (e.g. qemu) terminates.

This patch makes sure that all "leftover" protected VMs are always properly torn down.

So we're handling the kvm_arch_destroy_vm() case here, right?

yes

Quoted 3 lines

Maybe add that in a more prominent way and rework the subject:

KVM: s390: pv: cleanup leftover PV VM shells on VM shutdown

ok, I'll change the description and rework the subject

Quoted 43 lines
Quoted 43 lines
---
   arch/s390/include/asm/kvm_host.h |   2 +
   arch/s390/kvm/kvm-s390.c         |   2 +
   arch/s390/kvm/pv.c               | 109 ++++++++++++++++++++++++++++---
   3 files changed, 104 insertions(+), 9 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index 5824efe5fc9d..cca8e05e0a71 100644
@@ -924,6 +924,8 @@ struct kvm_s390_pv {
   	u64 guest_len;
   	unsigned long stor_base;
   	void *stor_var;
+	void *prepared_for_async_deinit;
+	struct list_head need_cleanup;
   	struct mmu_notifier mmu_notifier;
   };
 
arch/s390/kvm/kvm-s390.c +2 -0
index fe1fa896def7..369de8377116 100644
@@ -2890,6 +2890,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
   	kvm_s390_vsie_init(kvm);
   	if (use_gisa)
   		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.prepared_for_async_deinit = NULL;
   	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
   	return 0;
arch/s390/kvm/pv.c +0 -0
index 6cffea26c47f..8471c17d538c 100644
@@ -17,6 +17,19 @@
   #include <linux/mmu_notifier.h>
   #include "kvm-s390.h"
 
Quoted 4 lines

arch/s390/kvm/pv.c line 21

Quoted 2 lines
arch/s390/kvm/pv.c +2 -0
+/**
+ * @struct leftover_pv_vm

Any other ideas on naming these VMs?

not really

Quoted 1 line

Also I'd turn that around: pv_vm_leftover

I mean, it's a leftover protected VM, it felt more natural to name it that way

Quoted 16 lines

arch/s390/kvm/pv.c line 32

Quoted 12 lines
arch/s390/kvm/pv.c +11 -0
+ * Represents a "leftover" protected VM that is still registered with the
+ * Ultravisor, but which does not correspond any longer to an active KVM VM.
+ */
+struct leftover_pv_vm {
+	struct list_head list;
+	unsigned long old_gmap_table;
+	u64 handle;
+	void *stor_var;
+	unsigned long stor_base;
+};
+

I think we should switch this patch and the next one and add this struct to the next patch. The list work below makes more sense once the next patch has been read.

Quoted 5 lines

but the next patch will leave leftovers in some circumstances, and those won't be cleaned up without this patch.

having this patch first means that when the next patch is applied, the leftovers are already taken care of

Then I opt for squashing the patch.

Without the next patch prepared_for_async_deinit will always be NULL and this code is completely unneeded, no?

Quoted 12 lines
Quoted 11 lines
Quoted 10 lines
arch/s390/kvm/pv.c +0 -0
   static void kvm_s390_clear_pv_state(struct kvm *kvm)
   {
   	kvm->arch.pv.handle = 0;
@@ -158,23 +171,88 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
   	return -ENOMEM;
   }
 
 
 
author maintainer
#24 lore raw

Re: [PATCH v11 14/19] KVM: s390: pv: cleanup leftover protected VMs if needed

On Wed, 15 Jun 2022 12:57:39 +0200
Janosch Frank <[email protected]> wrote:

[...]

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I think we should switch this patch and the next one and add this struct to the next patch. The list work below makes more sense once the next patch has been read.

but the next patch will leave leftovers in some circumstances, and those won't be cleaned up without this patch.

Quoted 7 lines
Quoted 2 lines

having this patch first means that when the next patch is applied, the leftovers are already taken care of

Then I opt for squashing the patch.

Without the next patch prepared_for_async_deinit will always be NULL and this code is completely unneeded, no?

correct. I had split them to make them smaller and easier to review

I will squash them if you think it's better

Quoted 14 lines
Quoted 13 lines
Quoted 12 lines
Quoted 11 lines
  
   static void kvm_s390_clear_pv_state(struct kvm *kvm)
   {
   	kvm->arch.pv.handle = 0;
arch/s390/kvm/pv.c +0 -0
@@ -158,23 +171,88 @@ static int kvm_s390_pv_alloc_vm(struct kvm *kvm)
   	return -ENOMEM;
   }
 
 
 
author maintainer
#25 lore raw

PATCH v11 15/19 KVM: s390: pv: asynchronous destroy for reboot

In reply to Claudio Imbrenda

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_DISABLE_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail.

KVM_PV_ASYNC_DISABLE: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.

Leftover protected VMs are cleaned up when a KVM VM is torn down normally (either via IOCTL or when the process terminates); this cleanup has been implemented in a previous patch.

---
 arch/s390/kvm/kvm-s390.c |  34 +++++++++-
 arch/s390/kvm/kvm-s390.h |   2 +
 arch/s390/kvm/pv.c       | 131 +++++++++++++++++++++++++++++++++++++++
 include/uapi/linux/kvm.h |   2 +
 4 files changed, 166 insertions(+), 3 deletions(-)
arch/s390/kvm/kvm-s390.c +31 -3
index 369de8377116..842419092c0c 100644
@@ -2256,9 +2256,13 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
 
 static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 {
+	const bool needslock = (cmd->cmd != KVM_PV_ASYNC_DISABLE);
+	void __user *argp = (void __user *)cmd->data;
 	int r = 0;
 	u16 dummy;
-	void __user *argp = (void __user *)cmd->data;
+
+	if (needslock)
+		mutex_lock(&kvm->lock);
 
 	switch (cmd->cmd) {
 	case KVM_PV_ENABLE: {
@@ -2292,6 +2296,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 		set_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
 		break;
 	}
+	case KVM_PV_ASYNC_DISABLE_PREPARE:
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm) || !async_destroy)
+			break;
+
+		r = kvm_s390_cpus_from_pv(kvm, &cmd->rc, &cmd->rrc);
+		/*
+		 * If a CPU could not be destroyed, destroy VM will also fail.
+		 * There is no point in trying to destroy it. Instead return
+		 * the rc and rrc from the first CPU that failed destroying.
+		 */
+		if (r)
+			break;
+		r = kvm_s390_pv_deinit_vm_async_prepare(kvm, &cmd->rc, &cmd->rrc);
+
+		/* no need to block service interrupts any more */
+		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
+		break;
+	case KVM_PV_ASYNC_DISABLE:
+		/* This must not be called while holding kvm->lock */
+		r = kvm_s390_pv_deinit_vm_async(kvm, &cmd->rc, &cmd->rrc);
+		break;
 	case KVM_PV_DISABLE: {
 		r = -EINVAL;
 		if (!kvm_s390_pv_is_protected(kvm))
@@ -2393,6 +2419,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
 	default:
 		r = -ENOTTY;
 	}
+	if (needslock)
+		mutex_unlock(&kvm->lock);
+
 	return r;
 }
 
@@ -2597,9 +2626,8 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			r = -EINVAL;
 			break;
 		}
-		mutex_lock(&kvm->lock);
+		/* must be called without kvm->lock */
 		r = kvm_s390_handle_pv(kvm, &args);
-		mutex_unlock(&kvm->lock);
 		if (copy_to_user(argp, &args, sizeof(args))) {
 			r = -EFAULT;
 			break;
arch/s390/kvm/kvm-s390.h +2 -0
index d3abedafa7a8..d296afb6041c 100644
@@ -243,6 +243,8 @@ static inline u32 kvm_s390_get_gisa_desc(struct kvm *kvm)
 /* implemented in pv.c */
 int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
 int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async_prepare(struct kvm *kvm, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
arch/s390/kvm/pv.c +131 -0

137 lines not on this page · raw message

include/uapi/linux/kvm.h +2 -0
index 5088bd9f1922..91b072c137bf 100644
@@ -1668,6 +1668,8 @@ enum pv_cmd_id {
 	KVM_PV_VERIFY,
 	KVM_PV_PREP_RESET,
 	KVM_PV_UNSHARE_ALL,
+	KVM_PV_ASYNC_DISABLE_PREPARE,
+	KVM_PV_ASYNC_DISABLE,
 };
 
 struct kvm_pv_cmd {
Signature · 1 line
2.36.1
maintainer
#26 lore raw

Re: [PATCH v11 15/19] KVM: s390: pv: asynchronous destroy for reboot

Quoted 147 lines · Claudio Imbrenda

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_DISABLE_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail.

KVM_PV_ASYNC_DISABLE: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.

Leftover protected VMs are cleaned up when a KVM VM is torn down normally (either via IOCTL or when the process terminates); this cleanup has been implemented in a previous patch.

---
  arch/s390/kvm/kvm-s390.c |  34 +++++++++-
  arch/s390/kvm/kvm-s390.h |   2 +
  arch/s390/kvm/pv.c       | 131 +++++++++++++++++++++++++++++++++++++++
  include/uapi/linux/kvm.h |   2 +
  4 files changed, 166 insertions(+), 3 deletions(-)
arch/s390/kvm/kvm-s390.c +31 -3
index 369de8377116..842419092c0c 100644
@@ -2256,9 +2256,13 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
 
  static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  {
+	const bool needslock = (cmd->cmd != KVM_PV_ASYNC_DISABLE);
+	void __user *argp = (void __user *)cmd->data;
  	int r = 0;
  	u16 dummy;
-	void __user *argp = (void __user *)cmd->data;
+
+	if (needslock)
+		mutex_lock(&kvm->lock);
 
  	switch (cmd->cmd) {
  	case KVM_PV_ENABLE: {
@@ -2292,6 +2296,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  		set_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
  		break;
  	}
+	case KVM_PV_ASYNC_DISABLE_PREPARE:
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm) || !async_destroy)
+			break;
+
+		r = kvm_s390_cpus_from_pv(kvm, &cmd->rc, &cmd->rrc);
+		/*
+		 * If a CPU could not be destroyed, destroy VM will also fail.
+		 * There is no point in trying to destroy it. Instead return
+		 * the rc and rrc from the first CPU that failed destroying.
+		 */
+		if (r)
+			break;
+		r = kvm_s390_pv_deinit_vm_async_prepare(kvm, &cmd->rc, &cmd->rrc);
+
+		/* no need to block service interrupts any more */
+		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
+		break;
+	case KVM_PV_ASYNC_DISABLE:
+		/* This must not be called while holding kvm->lock */
+		r = kvm_s390_pv_deinit_vm_async(kvm, &cmd->rc, &cmd->rrc);
+		break;
  	case KVM_PV_DISABLE: {
  		r = -EINVAL;
  		if (!kvm_s390_pv_is_protected(kvm))
@@ -2393,6 +2419,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  	default:
  		r = -ENOTTY;
  	}
+	if (needslock)
+		mutex_unlock(&kvm->lock);
+
  	return r;
  }
 
@@ -2597,9 +2626,8 @@ long kvm_arch_vm_ioctl(struct file *filp,
  			r = -EINVAL;
  			break;
  		}
-		mutex_lock(&kvm->lock);
+		/* must be called without kvm->lock */
  		r = kvm_s390_handle_pv(kvm, &args);
-		mutex_unlock(&kvm->lock);
  		if (copy_to_user(argp, &args, sizeof(args))) {
  			r = -EFAULT;
  			break;
arch/s390/kvm/kvm-s390.h +2 -0
index d3abedafa7a8..d296afb6041c 100644
@@ -243,6 +243,8 @@ static inline u32 kvm_s390_get_gisa_desc(struct kvm *kvm)
  /* implemented in pv.c */
  int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
  int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async_prepare(struct kvm *kvm, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
arch/s390/kvm/pv.c +17 -0
index 8471c17d538c..ab06fa366e49 100644
@@ -279,6 +279,137 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
  	return cc ? -EIO : 0;
  }
 
+/**
+ * kvm_s390_destroy_lower_2g - Destroy the first 2GB of protected guest memory.
+ * @kvm the VM whose memory is to be cleared.
+ * Destroy the first 2GB of guest memory, to avoid prefix issues after reboot.
+ */
+static void kvm_s390_destroy_lower_2g(struct kvm *kvm)
+{
+	struct kvm_memory_slot *slot;
+	unsigned long lim;
+	int srcu_idx;
+
+	srcu_idx = srcu_read_lock(&kvm->srcu);
+
+	/* Take the memslot containing guest absolute address 0 */
+	slot = gfn_to_memslot(kvm, 0);
+	/* Clear all slots that are completely below 2GB */
+	while (slot && slot->base_gfn + slot->npages < SZ_2G / PAGE_SIZE) {

arch/s390/kvm/pv.c int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc) · line 308

Quoted 10 lines
arch/s390/kvm/pv.c +10 -0
+		lim = slot->userspace_addr + slot->npages * PAGE_SIZE;
+		s390_uv_destroy_range(kvm->mm, slot->userspace_addr, lim);
+		/* Take the next memslot */
+		slot = gfn_to_memslot(kvm, slot->base_gfn + slot->npages);
+	}
+	/* Last slot crosses the 2G boundary, clear only up to 2GB */
+	if (slot && slot->base_gfn < SZ_2G / PAGE_SIZE) {
+		lim = slot->userspace_addr + SZ_2G - slot->base_gfn * PAGE_SIZE;
+		s390_uv_destroy_range(kvm->mm, slot->userspace_addr, lim);
+	}

Any reason why you split that up instead of always calculating a length and using MIN()?

maintainer
#27 lore raw

Re: [PATCH v11 15/19] KVM: s390: pv: asynchronous destroy for reboot

In reply to Claudio Imbrenda
Quoted 72 lines · Claudio Imbrenda

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_DISABLE_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail.

KVM_PV_ASYNC_DISABLE: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.

Leftover protected VMs are cleaned up when a KVM VM is torn down normally (either via IOCTL or when the process terminates); this cleanup has been implemented in a previous patch.

---
  arch/s390/kvm/kvm-s390.c |  34 +++++++++-
  arch/s390/kvm/kvm-s390.h |   2 +
  arch/s390/kvm/pv.c       | 131 +++++++++++++++++++++++++++++++++++++++
  include/uapi/linux/kvm.h |   2 +
  4 files changed, 166 insertions(+), 3 deletions(-)
arch/s390/kvm/kvm-s390.c +14 -1
index 369de8377116..842419092c0c 100644
@@ -2256,9 +2256,13 @@ static int kvm_s390_cpus_to_pv(struct kvm *kvm, u16 *rc, u16 *rrc)
 
  static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  {
+	const bool needslock = (cmd->cmd != KVM_PV_ASYNC_DISABLE);
+	void __user *argp = (void __user *)cmd->data;
  	int r = 0;
  	u16 dummy;
-	void __user *argp = (void __user *)cmd->data;
+
+	if (needslock)
+		mutex_lock(&kvm->lock);
 
  	switch (cmd->cmd) {
  	case KVM_PV_ENABLE: {
@@ -2292,6 +2296,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  		set_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
  		break;
  	}
+	case KVM_PV_ASYNC_DISABLE_PREPARE:
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm) || !async_destroy)
+			break;
+
+		r = kvm_s390_cpus_from_pv(kvm, &cmd->rc, &cmd->rrc);
+		/*
+		 * If a CPU could not be destroyed, destroy VM will also fail.
+		 * There is no point in trying to destroy it. Instead return

arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2317

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10 -0
+		 * the rc and rrc from the first CPU that failed destroying.
+		 */
+		if (r)
+			break;
+		r = kvm_s390_pv_deinit_vm_async_prepare(kvm, &cmd->rc, &cmd->rrc);
+
+		/* no need to block service interrupts any more */
+		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
+		break;
+	case KVM_PV_ASYNC_DISABLE:

I'd like to have the ASYNC and DISABLE swapped so we're in line with the normal disable. Also renaming KVM_PV_ASYNC_DISABLE to KVM_PV_ASYNC_DISABLE_EXECUTE or something similar would make sense to me.

Quoted 45 lines
arch/s390/kvm/kvm-s390.c +7 -2
+		/* This must not be called while holding kvm->lock */
+		r = kvm_s390_pv_deinit_vm_async(kvm, &cmd->rc, &cmd->rrc);
+		break;
  	case KVM_PV_DISABLE: {
  		r = -EINVAL;
  		if (!kvm_s390_pv_is_protected(kvm))
@@ -2393,6 +2419,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  	default:
  		r = -ENOTTY;
  	}
+	if (needslock)
+		mutex_unlock(&kvm->lock);
+
  	return r;
  }
 
@@ -2597,9 +2626,8 @@ long kvm_arch_vm_ioctl(struct file *filp,
  			r = -EINVAL;
  			break;
  		}
-		mutex_lock(&kvm->lock);
+		/* must be called without kvm->lock */
  		r = kvm_s390_handle_pv(kvm, &args);
-		mutex_unlock(&kvm->lock);
  		if (copy_to_user(argp, &args, sizeof(args))) {
  			r = -EFAULT;
  			break;
arch/s390/kvm/kvm-s390.h +2 -0
index d3abedafa7a8..d296afb6041c 100644
@@ -243,6 +243,8 @@ static inline u32 kvm_s390_get_gisa_desc(struct kvm *kvm)
  /* implemented in pv.c */
  int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
  int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async_prepare(struct kvm *kvm, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_vm_async(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
  int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
arch/s390/kvm/pv.c +0 -0
index 8471c17d538c..ab06fa366e49 100644

arch/s390/kvm/pv.c int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc) · line 285

Quoted 7 lines
arch/s390/kvm/pv.c +4 -0
@@ -279,6 +279,137 @@ int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc)
  	return cc ? -EIO : 0;
  }
 
+/**
+ * kvm_s390_destroy_lower_2g - Destroy the first 2GB of protected guest memory.
+ * @kvm the VM whose memory is to be cleared.
+ * Destroy the first 2GB of guest memory, to avoid prefix issues after reboot.

Please add a line stating that destroying is only possible if the configuration that owns the storage has no PV cpus registered with the UV.

Quoted 31 lines
arch/s390/kvm/pv.c +31 -0
+ */
+static void kvm_s390_destroy_lower_2g(struct kvm *kvm)
+{
+	struct kvm_memory_slot *slot;
+	unsigned long lim;
+	int srcu_idx;
+
+	srcu_idx = srcu_read_lock(&kvm->srcu);
+
+	/* Take the memslot containing guest absolute address 0 */
+	slot = gfn_to_memslot(kvm, 0);
+	/* Clear all slots that are completely below 2GB */
+	while (slot && slot->base_gfn + slot->npages < SZ_2G / PAGE_SIZE) {
+		lim = slot->userspace_addr + slot->npages * PAGE_SIZE;
+		s390_uv_destroy_range(kvm->mm, slot->userspace_addr, lim);
+		/* Take the next memslot */
+		slot = gfn_to_memslot(kvm, slot->base_gfn + slot->npages);
+	}
+	/* Last slot crosses the 2G boundary, clear only up to 2GB */
+	if (slot && slot->base_gfn < SZ_2G / PAGE_SIZE) {
+		lim = slot->userspace_addr + SZ_2G - slot->base_gfn * PAGE_SIZE;
+		s390_uv_destroy_range(kvm->mm, slot->userspace_addr, lim);
+	}
+
+	srcu_read_unlock(&kvm->srcu, srcu_idx);
+}
+
+/**
+ * kvm_s390_pv_deinit_vm_async_prepare - Prepare a protected VM for
+ * asynchronous teardown.
+ * @kvm the VM

arch/s390/kvm/pv.c int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc) · line 326

Quoted 10 lines
arch/s390/kvm/pv.c +10 -0
+ * @rc return value for the RC field of the UVCB
+ * @rrc return value for the RRC field of the UVCB
+ *
+ * Prepare the protected VM for asynchronous teardown. The VM will be able
+ * to continue immediately as a non-secure VM, and the information needed to
+ * properly tear down the protected VM is set aside. If another protected VM
+ * was already set aside without starting a teardown, the function will
+ * fail.
+ *
+ * Context: kvm->lock needs to be held

You're asserting that via lockdep in the function below, why not here?

Also add something like: All PV cpus need to be destroyed by the caller before calling this function.

I'm considering that we add a cpu counter to the kvm arch pv struct which counts the currently created PV cpus so we can do a proper WARN_ONCE(count) here.

Quoted 37 lines
arch/s390/kvm/pv.c +37 -0
+ *
+ * Return: 0 in case of success, -EINVAL if another protected VM was already set
+ * aside, -ENOMEM if the system ran out of memory.
+ */
+int kvm_s390_pv_deinit_vm_async_prepare(struct kvm *kvm, u16 *rc, u16 *rrc)
+{
+	struct leftover_pv_vm *priv;
+
+	/*
+	 * If an asynchronous deinitialization is already pending, refuse.
+	 * A synchronous deinitialization has to be performed instead.
+	 */
+	if (READ_ONCE(kvm->arch.pv.prepared_for_async_deinit))
+		return -EINVAL;
+	priv = kmalloc(sizeof(*priv), GFP_KERNEL | __GFP_ZERO);
+	if (!priv)
+		return -ENOMEM;
+
+	priv->stor_var = kvm->arch.pv.stor_var;
+	priv->stor_base = kvm->arch.pv.stor_base;
+	priv->handle = kvm_s390_pv_get_handle(kvm);
+	priv->old_gmap_table = (unsigned long)kvm->arch.gmap->table;
+	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
+	if (s390_replace_asce(kvm->arch.gmap)) {
+		kfree(priv);
+		return -ENOMEM;
+	}
+
+	kvm_s390_destroy_lower_2g(kvm);
+	kvm_s390_clear_pv_state(kvm);
+	WRITE_ONCE(kvm->arch.pv.prepared_for_async_deinit, priv);
+
+	*rc = 1;
+	*rrc = 42;
+	return 0;
+}
+
author maintainer
#28 lore raw

PATCH v11 16/19 KVM: s390: pv: api documentation for asynchronous destroy

In reply to Claudio Imbrenda

Add documentation for the new commands added to the KVM_S390_PV_COMMAND ioctl.

---
 Documentation/virt/kvm/api.rst | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
Documentation/virt/kvm/api.rst +22 -3
index 11e00a46c610..97d35b30ce3b 100644
@@ -5143,11 +5143,13 @@ KVM_PV_ENABLE
   =====      =============================
 
 KVM_PV_DISABLE
-
   Deregister the VM from the Ultravisor and reclaim the memory that
   had been donated to the Ultravisor, making it usable by the kernel
-  again.  All registered VCPUs are converted back to non-protected
-  ones.
+  again. All registered VCPUs are converted back to non-protected
+  ones. If a previous VM had been prepared for asynchonous teardown
+  with KVM_PV_ASYNC_DISABLE_PREPARE and not actually torn down with
+  KVM_PV_ASYNC_DISABLE, it will be torn down in this call together with
+  the current VM.
 
 KVM_PV_VM_SET_SEC_PARMS
   Pass the image header from VM memory to the Ultravisor in
@@ -5160,6 +5162,23 @@ KVM_PV_VM_VERIFY
   Verify the integrity of the unpacked image. Only if this succeeds,
   KVM is allowed to start protected VCPUs.
 
+KVM_PV_ASYNC_DISABLE_PREPARE
+  Prepare the current protected VM for asynchronous teardown. Most
+  resources used by the current protected VM will be set aside for a
+  subsequent asynchronous teardown. The current protected VM will then
+  resume execution immediately as non-protected. If a protected VM had
+  already been prepared without starting the asynchronous teardown process,
+  this call will fail. In that case, the userspace process should issue a
+  normal KVM_PV_DISABLE.
+
+KVM_PV_ASYNC_DISABLE
+  Tear down the protected VM previously prepared for asynchronous teardown.
+  The resources that had been set aside will be freed asynchronously during
+  the execution of this command.
+  This PV command should ideally be issued by userspace from a separate
+  thread. If a fatal signal is received (or the process terminates
+  naturally), the command will terminate immediately without completing.
+
 4.126 KVM_X86_SET_MSR_FILTER
 ----------------------------
 
Signature · 1 line
2.36.1
maintainer
#29 lore raw

Re: [PATCH v11 16/19] KVM: s390: pv: api documentation for asynchronous destroy

Quoted 31 lines · Claudio Imbrenda

Add documentation for the new commands added to the KVM_S390_PV_COMMAND ioctl.

---
  Documentation/virt/kvm/api.rst | 25 ++++++++++++++++++++++---
  1 file changed, 22 insertions(+), 3 deletions(-)
Documentation/virt/kvm/api.rst +5 -3
index 11e00a46c610..97d35b30ce3b 100644
@@ -5143,11 +5143,13 @@ KVM_PV_ENABLE
    =====      =============================
 
  KVM_PV_DISABLE
-
    Deregister the VM from the Ultravisor and reclaim the memory that
    had been donated to the Ultravisor, making it usable by the kernel
-  again.  All registered VCPUs are converted back to non-protected
-  ones.
+  again. All registered VCPUs are converted back to non-protected
+  ones. If a previous VM had been prepared for asynchonous teardown
+  with KVM_PV_ASYNC_DISABLE_PREPARE and not actually torn down with
+  KVM_PV_ASYNC_DISABLE, it will be torn down in this call together with
+  the current VM.
 
  KVM_PV_VM_SET_SEC_PARMS
    Pass the image header from VM memory to the Ultravisor in

Documentation/virt/kvm/api.rst KVM_PV_VM_VERIFY · line 5167

Quoted 6 lines
Documentation/virt/kvm/api.rst +3 -0
@@ -5160,6 +5162,23 @@ KVM_PV_VM_VERIFY
    Verify the integrity of the unpacked image. Only if this succeeds,
    KVM is allowed to start protected VCPUs.
 
+KVM_PV_ASYNC_DISABLE_PREPARE
+  Prepare the current protected VM for asynchronous teardown. Most
+  resources used by the current protected VM will be set aside for a

We should state that leftover UV state needs cleanup, namely secure storage and the configuration.

author maintainer
#30 lore raw

PATCH v11 17/19 KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE

In reply to Claudio Imbrenda

Add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE to signal that the KVM_PV_ASYNC_DISABLE and KVM_PV_ASYNC_DISABLE_PREPARE commands for the KVM_S390_PV_COMMAND ioctl are available.

---
 arch/s390/kvm/kvm-s390.c | 3 +++
 include/uapi/linux/kvm.h | 1 +
 2 files changed, 4 insertions(+)
arch/s390/kvm/kvm-s390.c +3 -0
index 842419092c0c..aeb654849f2d 100644
@@ -609,6 +609,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_S390_BPB:
 		r = test_facility(82);
 		break;
+	case KVM_CAP_S390_PROTECTED_ASYNC_DISABLE:
+		r = async_destroy && is_prot_virt_host();
+		break;
 	case KVM_CAP_S390_PROTECTED:
 		r = is_prot_virt_host();
 		break;
include/uapi/linux/kvm.h +1 -0
index 91b072c137bf..b22dd1221b30 100644
@@ -1157,6 +1157,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_VM_TSC_CONTROL 214
 #define KVM_CAP_SYSTEM_EVENT_DATA 215
 #define KVM_CAP_ARM_SYSTEM_SUSPEND 216
+#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 225
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.36.1
author maintainer
#31 lore raw

PATCH v11 18/19 KVM: s390: pv: avoid export before import if possible

In reply to Claudio Imbrenda

If the appropriate UV feature bit is set, there is no need to perform an export before import.

---
 arch/s390/kernel/uv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/s390/kernel/uv.c +2 -0
index 02aca3c5dce1..c18c3d6a4314 100644
@@ -249,6 +249,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,
  */
 static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
 {
+	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
+		return false;
 	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
 		return false;
 	return atomic_read(&mm->context.protected_count) > 1;
Signature · 1 line
2.36.1
maintainer
#33 lore raw

Re: [PATCH v11 18/19] KVM: s390: pv: avoid export before import if possible

In reply to Claudio Imbrenda
Quoted 13 lines · Claudio Imbrenda

If the appropriate UV feature bit is set, there is no need to perform an export before import.

---
  arch/s390/kernel/uv.c | 2 ++
  1 file changed, 2 insertions(+)
arch/s390/kernel/uv.c +0 -0
index 02aca3c5dce1..c18c3d6a4314 100644

arch/s390/kernel/uv.c static int make_secure_pte(pte_t *ptep, unsigned long addr, · line 253

Quoted 5 lines
arch/s390/kernel/uv.c +2 -0
@@ -249,6 +249,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,
   */
  static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
  {
+	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
+		return false;

This needs a comment explaining, that this is only an option for shared pages.

Quoted 3 lines
arch/s390/kernel/uv.c +0 -0
  	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
  		return false;
  	return atomic_read(&mm->context.protected_count) > 1;
author maintainer
#34 lore raw

PATCH v11 19/19 KVM: s390: pv: support for Destroy fast UVC

In reply to Claudio Imbrenda

Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.

When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.

---
 arch/s390/include/asm/uv.h | 10 +++++++
 arch/s390/kvm/pv.c         | 58 ++++++++++++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 9 deletions(-)
arch/s390/include/asm/uv.h +10 -0
index ba64e0be03bb..8b255d26c5a7 100644
@@ -34,6 +34,7 @@
 #define UVC_CMD_INIT_UV			0x000f
 #define UVC_CMD_CREATE_SEC_CONF		0x0100
 #define UVC_CMD_DESTROY_SEC_CONF	0x0101
+#define UVC_CMD_DESTROY_SEC_CONF_FAST	0x0102
 #define UVC_CMD_CREATE_SEC_CPU		0x0120
 #define UVC_CMD_DESTROY_SEC_CPU		0x0121
 #define UVC_CMD_CONV_TO_SEC_STOR	0x0200
@@ -77,6 +78,7 @@ enum uv_cmds_inst {
 	BIT_UVC_CMD_UNSHARE_ALL = 20,
 	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
 	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
+	BIT_UVC_CMD_DESTROY_SEC_CONF_FAST = 23,
 	BIT_UVC_CMD_RETR_ATTEST = 28,
 };
 
@@ -213,6 +215,14 @@ struct uv_cb_nodata {
 	u64 reserved20[4];
 } __packed __aligned(8);
 
+/* Destroy Configuration Fast */
+struct uv_cb_destroy_fast {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 handle;
+	u64 reserved20[5];
+} __packed __aligned(8);
+
 /* Set Shared Access */
 struct uv_cb_share {
 	struct uv_cb_header header;
arch/s390/kvm/pv.c +49 -9

100 lines not on this page · raw message

Signature · 1 line
2.36.1
maintainer
#35 lore raw
In reply to Claudio Imbrenda
Quoted 7 lines · Claudio Imbrenda

Previously, when a protected VM was rebooted or when it was shut down, its memory was made unprotected, and then the protected VM itself was destroyed. Looping over the whole address space can take some time, considering the overhead of the various Ultravisor Calls (UVCs). This means that a reboot or a shutdown would take a potentially long amount of time, depending on the amount of used memory.

Quoted 9 lines

This patchseries implements a deferred destroy mechanism for protected guests. When a protected guest is destroyed, its memory can be cleared in background, allowing the guest to restart or terminate significantly faster than before.

There are 2 possibilities when a protected VM is torn down:
* it still has an address space associated (reboot case)
* it does not have an address space anymore (shutdown case)

Please add patches 1-13 to devel for some CI coverage. I'll try reviewing the remaining patches this week.