Skip to contentSkip to conversation

KVM: s390: pv: implement lazy destroy for reboot

PATCH Superseded Claudio Imbrenda 3 participants 14 messages Aug 10, 2022 – Aug 19, 2022

Subsystems (4)

v13 of 16 Superseded Latest is v16 →

lore
Share & tools
lore

Claudio Imbrenda

author maintainer
#0 lore raw

v13 cover KVM: s390: pv: implement lazy destroy for reboot

Previously, when a protected VM was rebooted or when it was shut down, its memory was made unprotected, and then the protected VM itself was destroyed. Looping over the whole address space can take some time, considering the overhead of the various Ultravisor Calls (UVCs). This means that a reboot or a shutdown would take a potentially long amount of time, depending on the amount of used memory.

This patchseries implements a deferred destroy mechanism for protected guests. When a protected guest is destroyed, its memory can be cleared in background, allowing the guest to restart or terminate significantly faster than before.

There are 2 possibilities when a protected VM is torn down:
* it still has an address space associated (reboot case)
* it does not have an address space anymore (shutdown case)

For the reboot case, two new commands are available for the KVM_S390_PV_COMMAND:

KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. In this case the userspace process should issue a normal KVM_PV_DISABLE

KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asychronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all leftovers.

The idea is that userspace should first issue the KVM_PV_ASYNC_CLEANUP_PREPARE command, and in case of success, create a new thread and issue KVM_PV_ASYNC_CLEANUP_PERFORM from there. This also allows for proper accounting of the CPU time needed for the asynchronous teardown.

This means that the same address space can have memory belonging to more than one protected guest, although only one will be running, the others will in fact not even have any CPUs.

The shutdown case should be dealt with in userspace (e.g. using clone(CLONE_VM)).

A module parameter is also provided to disable the new functionality, which is otherwise enabled by default. This should not be an issue since the new functionality is opt-in anyway. This is mainly thought to aid debugging.

v12->v13
* drop the patches that have been already merged
* rebase

Claudio Imbrenda (6):
KVM: s390: pv: asynchronous destroy for reboot
KVM: s390: pv: api documentation for asynchronous destroy
KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
KVM: s390: pv: avoid export before import if possible
KVM: s390: pv: support for Destroy fast UVC
KVM: s390: pv: module parameter to fence asynchronous destroy

 Documentation/virt/kvm/api.rst   |  30 ++-
 arch/s390/include/asm/kvm_host.h |   2 +
 arch/s390/include/asm/uv.h       |  10 +
 arch/s390/kernel/uv.c            |   2 +
 arch/s390/kvm/kvm-s390.c         |  55 ++++-
 arch/s390/kvm/kvm-s390.h         |   3 +
 arch/s390/kvm/pv.c               | 331 ++++++++++++++++++++++++++++++-
 include/uapi/linux/kvm.h         |   3 +
 8 files changed, 416 insertions(+), 20 deletions(-)
Signature · 1 line
2.37.1
author maintainer
#1 lore raw

PATCH v13 1/6 KVM: s390: pv: asynchronous destroy for reboot

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.

KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all remaining protected VMs.

---
 arch/s390/include/asm/kvm_host.h |   2 +
 arch/s390/kvm/kvm-s390.c         |  46 ++++-
 arch/s390/kvm/kvm-s390.h         |   3 +
 arch/s390/kvm/pv.c               | 290 +++++++++++++++++++++++++++++--
 include/uapi/linux/kvm.h         |   2 +
 5 files changed, 325 insertions(+), 18 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index f39092e0ceaa..38558e5f554e 100644
@@ -942,6 +942,8 @@ struct kvm_s390_pv {
 	unsigned long stor_base;
 	void *stor_var;
 	bool dumping;
+	void *set_aside;
+	struct list_head need_cleanup;
 	struct mmu_notifier mmu_notifier;
 };
 
arch/s390/kvm/kvm-s390.c +38 -8

98 lines not on this page · raw message

arch/s390/kvm/kvm-s390.h +3 -0
index f6fd668f887e..e70b895f640e 100644
@@ -243,6 +243,9 @@ static inline u32 kvm_s390_get_gisa_desc(struct kvm *kvm)
 /* implemented in pv.c */
 int kvm_s390_pv_destroy_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
 int kvm_s390_pv_create_cpu(struct kvm_vcpu *vcpu, u16 *rc, u16 *rrc);
+int kvm_s390_pv_set_aside(struct kvm *kvm, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_aside_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
+int kvm_s390_pv_deinit_cleanup_all(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_deinit_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_init_vm(struct kvm *kvm, u16 *rc, u16 *rrc);
 int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
arch/s390/kvm/pv.c +280 -10

315 lines not on this page · raw message

include/uapi/linux/kvm.h +2 -0
index eed0315a77a6..02602c5c1975 100644
@@ -1739,6 +1739,8 @@ enum pv_cmd_id {
 	KVM_PV_UNSHARE_ALL,
 	KVM_PV_INFO,
 	KVM_PV_DUMP,
+	KVM_PV_ASYNC_CLEANUP_PREPARE,
+	KVM_PV_ASYNC_CLEANUP_PERFORM,
 };
 
 struct kvm_pv_cmd {
Signature · 1 line
2.37.1
maintainer
#2 lore raw

Re: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot

Quoted 95 lines · Claudio Imbrenda

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.

KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all remaining protected VMs.

---
  arch/s390/include/asm/kvm_host.h |   2 +
  arch/s390/kvm/kvm-s390.c         |  46 ++++-
  arch/s390/kvm/kvm-s390.h         |   3 +
  arch/s390/kvm/pv.c               | 290 +++++++++++++++++++++++++++++--
  include/uapi/linux/kvm.h         |   2 +
  5 files changed, 325 insertions(+), 18 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
index f39092e0ceaa..38558e5f554e 100644
@@ -942,6 +942,8 @@ struct kvm_s390_pv {
  	unsigned long stor_base;
  	void *stor_var;
  	bool dumping;
+	void *set_aside;
+	struct list_head need_cleanup;
  	struct mmu_notifier mmu_notifier;
  };
 
arch/s390/kvm/kvm-s390.c +17 -1
index edfd4bbd0cba..96ec6ecb6b74 100644
@@ -209,6 +209,8 @@ unsigned int diag9c_forwarding_hz;
  module_param(diag9c_forwarding_hz, uint, 0644);
  MODULE_PARM_DESC(diag9c_forwarding_hz, "Maximum diag9c forwarding per second, 0 to turn off");
 
+static int async_destroy;
+
  /*
   * For now we handle at most 16 double words as this is what the s390 base
   * kernel handles and stores in the prefix page. If we ever need to go beyond
@@ -2504,9 +2506,13 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,
 
  static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  {
+	const bool needslock = (cmd->cmd != KVM_PV_ASYNC_CLEANUP_PERFORM);
+	void __user *argp = (void __user *)cmd->data;
  	int r = 0;
  	u16 dummy;
-	void __user *argp = (void __user *)cmd->data;
+
+	if (needslock)
+		mutex_lock(&kvm->lock);
 
  	switch (cmd->cmd) {
  	case KVM_PV_ENABLE: {
@@ -2540,6 +2546,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  		set_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
  		break;
  	}
+	case KVM_PV_ASYNC_CLEANUP_PREPARE:
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm) || !async_destroy)
+			break;
+
+		r = kvm_s390_cpus_from_pv(kvm, &cmd->rc, &cmd->rrc);
+		/*
+		 * If a CPU could not be destroyed, destroy VM will also fail.
+		 * There is no point in trying to destroy it. Instead return
+		 * the rc and rrc from the first CPU that failed destroying.

arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2568

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10 -0
+		 */
+		if (r)
+			break;
+		r = kvm_s390_pv_set_aside(kvm, &cmd->rc, &cmd->rrc);
+
+		/* no need to block service interrupts any more */
+		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
+		break;
+	case KVM_PV_ASYNC_CLEANUP_PERFORM:
+		/* This must not be called while holding kvm->lock */

Two things:
I know that we don't need to check async_destroy since it will find nothing to cleanup because the command above is fenced. But I'd still appreciate the same check here.

Consider adding this to the comment:
", this is asserted inside the function."

Quoted 25 lines
arch/s390/kvm/kvm-s390.c +6 -1
+		r = kvm_s390_pv_deinit_aside_vm(kvm, &cmd->rc, &cmd->rrc);
+		break;
  	case KVM_PV_DISABLE: {
  		r = -EINVAL;
  		if (!kvm_s390_pv_is_protected(kvm))
@@ -2553,7 +2581,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  		 */
  		if (r)
  			break;
-		r = kvm_s390_pv_deinit_vm(kvm, &cmd->rc, &cmd->rrc);
+		r = kvm_s390_pv_deinit_cleanup_all(kvm, &cmd->rc, &cmd->rrc);
 
  		/* no need to block service interrupts any more */
  		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
@@ -2703,6 +2731,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  	default:
  		r = -ENOTTY;
  	}
+	if (needslock)
+		mutex_unlock(&kvm->lock);
+
  	return r;
  }
 

arch/s390/kvm/kvm-s390.c long kvm_arch_vm_ioctl(struct file *filp, · line 2941

Quoted 5 lines
arch/s390/kvm/kvm-s390.c +1 -1
@@ -2907,9 +2938,8 @@ long kvm_arch_vm_ioctl(struct file *filp,
  			r = -EINVAL;
  			break;
  		}
-		mutex_lock(&kvm->lock);
+		/* must be called without kvm->lock */

...as it will acquire and release it by itself.

Quoted 28 lines
arch/s390/kvm/kvm-s390.c +4 -5
  		r = kvm_s390_handle_pv(kvm, &args);
-		mutex_unlock(&kvm->lock);
  		if (copy_to_user(argp, &args, sizeof(args))) {
  			r = -EFAULT;
  			break;
@@ -3228,6 +3258,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  	kvm_s390_vsie_init(kvm);
  	if (use_gisa)
  		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.set_aside = NULL;
  	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
  	return 0;
@@ -3272,11 +3304,9 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
  	/*
  	 * We are already at the end of life and kvm->lock is not taken.
  	 * This is ok as the file descriptor is closed by now and nobody
-	 * can mess with the pv state. To avoid lockdep_assert_held from
-	 * complaining we do not use kvm_s390_pv_is_protected.
+	 * can mess with the pv state.
  	 */
-	if (kvm_s390_pv_get_handle(kvm))
-		kvm_s390_pv_deinit_vm(kvm, &rc, &rrc);
+	kvm_s390_pv_deinit_cleanup_all(kvm, &rc, &rrc);
  	/*
  	 * Remove the mmu notifier only when the whole KVM VM is torn down,
  	 * and only if one was registered to begin with. If the VM is

[...]

Quoted 15 lines
arch/s390/kvm/kvm-s390.c +15 -0
+
+/**
+ * kvm_s390_pv_set_aside - Set aside a protected VM for later teardown.
+ * @kvm: the VM
+ * @rc: return value for the RC field of the UVCB
+ * @rrc: return value for the RRC field of the UVCB
+ *
+ * Set aside the protected VM for a subsequent teardown. The VM will be able
+ * to continue immediately as a non-secure VM, and the information needed to
+ * properly tear down the protected VM is set aside. If another protected VM
+ * was already set aside without starting its teardown, this function will
+ * fail.
+ * The CPUs of the protected VM need to be destroyed beforehand.
+ *
+ * Context: kvm->lock needs to be held

arch/s390/kvm/kvm-s390.c

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10 -0
+ *
+ * Return: 0 in case of success, -EINVAL if another protected VM was already set
+ * aside, -ENOMEM if the system ran out of memory.
+ */
+int kvm_s390_pv_set_aside(struct kvm *kvm, u16 *rc, u16 *rrc)
+{
+	struct pv_vm_to_be_destroyed *priv;
+
+	/*
+	 * If another protected VM was already prepared, refuse.

s/prepared/set aside/
or
prepared for teardown

arch/s390/kvm/kvm-s390.c

Quoted 5 lines
arch/s390/kvm/kvm-s390.c +5 -0
+	 * A normal deinitialization has to be performed instead.
+	 */
+	if (kvm->arch.pv.set_aside)
+		return -EINVAL;
+	priv = kmalloc(sizeof(*priv), GFP_KERNEL | __GFP_ZERO);

kzalloc()?

Quoted 8 lines
arch/s390/kvm/kvm-s390.c +8 -0
+	if (!priv)
+		return -ENOMEM;
+
+	priv->stor_var = kvm->arch.pv.stor_var;
+	priv->stor_base = kvm->arch.pv.stor_base;
+	priv->handle = kvm_s390_pv_get_handle(kvm);
+	priv->old_gmap_table = (unsigned long)kvm->arch.gmap->table;
+	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);

arch/s390/kvm/kvm-s390.c

Quoted 10 lines
arch/s390/kvm/kvm-s390.c +8 -0
+	if (s390_replace_asce(kvm->arch.gmap)) {
+		kfree(priv);
+		return -ENOMEM;
  	}
 
+	kvm_s390_destroy_lower_2g(kvm);
+	kvm_s390_clear_pv_state(kvm);
+	kvm->arch.pv.set_aside = priv;
+
+	*rc = 1;

UVC_RC_EXECUTED

arch/s390/kvm/kvm-s390.c

Quoted 1 line
arch/s390/kvm/kvm-s390.c +1 -0
+	*rrc = 42;

I'd prefer setting the rrc to 0.

Quoted 2 lines
arch/s390/kvm/kvm-s390.c +2 -0
+	return 0;
+}
author maintainer
#3 lore raw

Re: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot

On Thu, 11 Aug 2022 18:26:13 +0200
Janosch Frank <[email protected]> wrote:

[...]

Quoted 17 lines
Quoted 17 lines
arch/s390/kvm/kvm-s390.c +17 -0
+	case KVM_PV_ASYNC_CLEANUP_PREPARE:
+		r = -EINVAL;
+		if (!kvm_s390_pv_is_protected(kvm) || !async_destroy)
+			break;
+
+		r = kvm_s390_cpus_from_pv(kvm, &cmd->rc, &cmd->rrc);
+		/*
+		 * If a CPU could not be destroyed, destroy VM will also fail.
+		 * There is no point in trying to destroy it. Instead return
+		 * the rc and rrc from the first CPU that failed destroying.
+		 */
+		if (r)
+			break;
+		r = kvm_s390_pv_set_aside(kvm, &cmd->rc, &cmd->rrc);
+
+		/* no need to block service interrupts any more */
+		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
Quoted 8 lines

arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2568

Quoted 3 lines
arch/s390/kvm/kvm-s390.c +3 -0
+		break;
+	case KVM_PV_ASYNC_CLEANUP_PERFORM:
+		/* This must not be called while holding kvm->lock */  

Two things:
I know that we don't need to check async_destroy since it will find nothing to cleanup because the command above is fenced. But I'd still appreciate the same check here.

will add

Quoted 3 lines

Consider adding this to the comment:
", this is asserted inside the function."

will add

Quoted 25 lines
Quoted 25 lines
arch/s390/kvm/kvm-s390.c +6 -1
+		r = kvm_s390_pv_deinit_aside_vm(kvm, &cmd->rc, &cmd->rrc);
+		break;
  	case KVM_PV_DISABLE: {
  		r = -EINVAL;
  		if (!kvm_s390_pv_is_protected(kvm))
@@ -2553,7 +2581,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  		 */
  		if (r)
  			break;
-		r = kvm_s390_pv_deinit_vm(kvm, &cmd->rc, &cmd->rrc);
+		r = kvm_s390_pv_deinit_cleanup_all(kvm, &cmd->rc, &cmd->rrc);
 
  		/* no need to block service interrupts any more */
  		clear_bit(IRQ_PEND_EXT_SERVICE, &kvm->arch.float_int.masked_irqs);
@@ -2703,6 +2731,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)
  	default:
  		r = -ENOTTY;
  	}
+	if (needslock)
+		mutex_unlock(&kvm->lock);
+
  	return r;
  }
 
Quoted 8 lines

arch/s390/kvm/kvm-s390.c long kvm_arch_vm_ioctl(struct file *filp, · line 2941

Quoted 6 lines
arch/s390/kvm/kvm-s390.c +1 -1
@@ -2907,9 +2938,8 @@ long kvm_arch_vm_ioctl(struct file *filp,
  			r = -EINVAL;
  			break;
  		}
-		mutex_lock(&kvm->lock);
+		/* must be called without kvm->lock */  

...as it will acquire and release it by itself.

none of the other switch cases acquire kvm->lock, I actually think the comment is redundant as it is, I don't think we need to expand it further.

Quoted 51 lines
Quoted 29 lines
arch/s390/kvm/kvm-s390.c +4 -5
  		r = kvm_s390_handle_pv(kvm, &args);
-		mutex_unlock(&kvm->lock);
  		if (copy_to_user(argp, &args, sizeof(args))) {
  			r = -EFAULT;
  			break;
@@ -3228,6 +3258,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  	kvm_s390_vsie_init(kvm);
  	if (use_gisa)
  		kvm_s390_gisa_init(kvm);
+	INIT_LIST_HEAD(&kvm->arch.pv.need_cleanup);
+	kvm->arch.pv.set_aside = NULL;
  	KVM_EVENT(3, "vm 0x%pK created by pid %u", kvm, current->pid);
 
  	return 0;
@@ -3272,11 +3304,9 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
  	/*
  	 * We are already at the end of life and kvm->lock is not taken.
  	 * This is ok as the file descriptor is closed by now and nobody
-	 * can mess with the pv state. To avoid lockdep_assert_held from
-	 * complaining we do not use kvm_s390_pv_is_protected.
+	 * can mess with the pv state.
  	 */
-	if (kvm_s390_pv_get_handle(kvm))
-		kvm_s390_pv_deinit_vm(kvm, &rc, &rrc);
+	kvm_s390_pv_deinit_cleanup_all(kvm, &rc, &rrc);
  	/*
  	 * Remove the mmu notifier only when the whole KVM VM is torn down,
  	 * and only if one was registered to begin with. If the VM is  

[...]

Quoted 21 lines
arch/s390/kvm/pv.c +21 -0
+
+/**
+ * kvm_s390_pv_set_aside - Set aside a protected VM for later teardown.
+ * @kvm: the VM
+ * @rc: return value for the RC field of the UVCB
+ * @rrc: return value for the RRC field of the UVCB
+ *
+ * Set aside the protected VM for a subsequent teardown. The VM will be able
+ * to continue immediately as a non-secure VM, and the information needed to
+ * properly tear down the protected VM is set aside. If another protected VM
+ * was already set aside without starting its teardown, this function will
+ * fail.
+ * The CPUs of the protected VM need to be destroyed beforehand.
+ *
+ * Context: kvm->lock needs to be held
+ *
+ * Return: 0 in case of success, -EINVAL if another protected VM was already set
+ * aside, -ENOMEM if the system ran out of memory.
+ */
+int kvm_s390_pv_set_aside(struct kvm *kvm, u16 *rc, u16 *rrc)
+{
Quoted 8 lines

arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 274

Quoted 4 lines
arch/s390/kvm/pv.c +4 -0
+	struct pv_vm_to_be_destroyed *priv;
+
+	/*
+	 * If another protected VM was already prepared, refuse.  

s/prepared/set aside/
or
prepared for teardown

prepared for teardown; will fix

Quoted 8 lines

arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 279

Quoted 6 lines
arch/s390/kvm/pv.c +5 -0
+	 * A normal deinitialization has to be performed instead.
+	 */
+	if (kvm->arch.pv.set_aside)
+		return -EINVAL;
+	priv = kmalloc(sizeof(*priv), GFP_KERNEL | __GFP_ZERO);  

kzalloc()?

oops, yes

Quoted 14 lines
Quoted 14 lines
arch/s390/kvm/pv.c +11 -0
+	if (!priv)
+		return -ENOMEM;
+
+	priv->stor_var = kvm->arch.pv.stor_var;
+	priv->stor_base = kvm->arch.pv.stor_base;
+	priv->handle = kvm_s390_pv_get_handle(kvm);
+	priv->old_gmap_table = (unsigned long)kvm->arch.gmap->table;
+	WRITE_ONCE(kvm->arch.gmap->guest_handle, 0);
+	if (s390_replace_asce(kvm->arch.gmap)) {
+		kfree(priv);
+		return -ENOMEM;
  	}
Quoted 7 lines

arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 297

Quoted 5 lines
arch/s390/kvm/pv.c +5 -0
+	kvm_s390_destroy_lower_2g(kvm);
+	kvm_s390_clear_pv_state(kvm);
+	kvm->arch.pv.set_aside = priv;
+
+	*rc = 1;  

UVC_RC_EXECUTED

will fix

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +1 -0
+	*rrc = 42;  

I'd prefer setting the rrc to 0.

I'd like to convey the information that the "successful" execution was actually faked

Quoted 3 lines
Quoted 3 lines
(quoted diff) +2 -0
+	return 0;
+}  
Reviewed-by on v13 1/6
#4 lore raw

Re: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot

In reply to Claudio Imbrenda

Quoting Claudio Imbrenda (2022-08-10 14:56:20)

Quoted 12 lines

Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.

This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.

This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.

Quoted 7 lines

The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.

Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:

KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown.

I would add:

A protected VM which has been prepared for asynchronous teardown is called a *set aside VM*. A set a side VM never has any CPUs associated with it, but is still registered with the Ultravisor and may have memory.

Quoted 10 lines

The current VM will then continue immediately
as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.

KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.

I would add:

A set aside VM where cleanup started but was interrupted is called a *leftover VM*. There can be multiple leftovers per VM, which are tracked in the need_cleanup list.

I would like a more consistent language. We have three different names for the leftover in the patch below:
- leftover,
- the struct describing a leftover is called pv_vm_to_be_destroyed,
- the list of leftover vms is called need_cleanup.

It would be nice if this were a little more consistent, e.g. by renaming pv_vm_to_be_destroyed to pvm_vm_leftover and maybe need_cleanup to leftovers.

Otherwise, this looks good. I would be glad if you pick up my naming suggestions, but feel free to add:

author maintainer
#5 lore raw

PATCH v13 2/6 KVM: s390: pv: api documentation for asynchronous destroy

In reply to Claudio Imbrenda

Add documentation for the new commands added to the KVM_S390_PV_COMMAND ioctl.

---
 Documentation/virt/kvm/api.rst | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +28 -2
index 9788b19f9ff7..5bd151b601b4 100644
@@ -5163,8 +5163,11 @@ KVM_PV_ENABLE
 KVM_PV_DISABLE
   Deregister the VM from the Ultravisor and reclaim the memory that
   had been donated to the Ultravisor, making it usable by the kernel
-  again.  All registered VCPUs are converted back to non-protected
-  ones.
+  again. All registered VCPUs are converted back to non-protected
+  ones. If a previous VM had been set aside for asynchonous teardown
+  with KVM_PV_ASYNC_CLEANUP_PREPARE and not actually torn down with
+  KVM_PV_ASYNC_CLEANUP_PERFORM, it will be torn down in this call
+  together with the current VM.
 
 KVM_PV_VM_SET_SEC_PARMS
   Pass the image header from VM memory to the Ultravisor in
@@ -5287,6 +5290,29 @@ KVM_PV_DUMP
     authentication tag all of which are needed to decrypt the dump at a
     later time.
 
+KVM_PV_ASYNC_CLEANUP_PREPARE
+  Prepare the current protected VM for asynchronous teardown. Most
+  resources used by the current protected VM will be set aside for a
+  subsequent asynchronous teardown. The current protected VM will then
+  resume execution immediately as non-protected. There can be at most
+  one protected VM set aside at any time. If a protected VM had
+  already been set aside without starting the asynchronous teardown
+  process, this call will fail. In that case, the userspace process
+  should issue a normal KVM_PV_DISABLE. The resources set aside with
+  this call will need to be cleaned up with a subsequent call to
+  KVM_PV_ASYNC_CLEANUP_PERFORM or KVM_PV_DISABLE, otherwise they will
+  be cleaned up when KVM terminates.
+
+KVM_PV_ASYNC_CLEANUP_PERFORM
+  Tear down the protected VM previously set aside with
+  KVM_PV_ASYNC_CLEANUP_PREPARE. The resources that had been set aside
+  will be freed during the execution of this command. This PV command
+  should ideally be issued by userspace from a separate thread. If a
+  fatal signal is received (or the process terminates naturally), the
+  command will terminate immediately without completing, and the normal
+  KVM shutdown procedure will take care of cleaning up all remaining
+  protected VMs.
+
 
 4.126 KVM_X86_SET_MSR_FILTER
 ----------------------------
Signature · 1 line
2.37.1
maintainer
#6 lore raw

Re: [PATCH v13 2/6] KVM: s390: pv: api documentation for asynchronous destroy

Quoted 14 lines · Claudio Imbrenda

Add documentation for the new commands added to the KVM_S390_PV_COMMAND ioctl.

---
  Documentation/virt/kvm/api.rst | 30 ++++++++++++++++++++++++++++--
  1 file changed, 28 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +0 -0
index 9788b19f9ff7..5bd151b601b4 100644

Documentation/virt/kvm/api.rst KVM_PV_ENABLE · line 5168

Quoted 8 lines
Documentation/virt/kvm/api.rst +3 -2
@@ -5163,8 +5163,11 @@ KVM_PV_ENABLE
  KVM_PV_DISABLE
    Deregister the VM from the Ultravisor and reclaim the memory that
    had been donated to the Ultravisor, making it usable by the kernel
-  again.  All registered VCPUs are converted back to non-protected
-  ones.
+  again. All registered VCPUs are converted back to non-protected
+  ones. If a previous VM had been set aside for asynchonous teardown
+  with KVM_PV_ASYNC_CLEANUP_PREPARE and not actually torn down with

...and hasn't yet been torn down with...

Documentation/virt/kvm/api.rst KVM_PV_ENABLE · line 5170

Quoted 2 lines
Documentation/virt/kvm/api.rst +2 -0
+  KVM_PV_ASYNC_CLEANUP_PERFORM, it will be torn down in this call
+  together with the current VM.

current PV VM? or protected VM
I know it's missing in the unchanged paragraph above too but such is life.

Quoted 5 lines
Documentation/virt/kvm/api.rst +0 -0
  KVM_PV_VM_SET_SEC_PARMS
    Pass the image header from VM memory to the Ultravisor in
@@ -5287,6 +5290,29 @@ KVM_PV_DUMP
      authentication tag all of which are needed to decrypt the dump at a

Documentation/virt/kvm/api.rst KVM_PV_DUMP · line 5300

Quoted 10 lines
Documentation/virt/kvm/api.rst +8 -0
      later time.
 
+KVM_PV_ASYNC_CLEANUP_PREPARE
+  Prepare the current protected VM for asynchronous teardown. Most
+  resources used by the current protected VM will be set aside for a
+  subsequent asynchronous teardown. The current protected VM will then
+  resume execution immediately as non-protected. There can be at most
+  one protected VM set aside at any time. If a protected VM had
+  already been set aside without starting the asynchronous teardown
+  process, this call will fail. In that case, the userspace process

If KVM_PV_ASYNC_CLEANUP_PREPARE has already been called without a successful KVM_PV_ASYNC_CLEANUP_PERFORM this call will fail. I.e. only be one PV VM can be set aside.

Do we need to finish the cleanup or is it enough to start the cleanup like you describe here?

Quoted 18 lines
Documentation/virt/kvm/api.rst +15 -0
+  should issue a normal KVM_PV_DISABLE. The resources set aside with
+  this call will need to be cleaned up with a subsequent call to
+  KVM_PV_ASYNC_CLEANUP_PERFORM or KVM_PV_DISABLE, otherwise they will
+  be cleaned up when KVM terminates.
+
+KVM_PV_ASYNC_CLEANUP_PERFORM
+  Tear down the protected VM previously set aside with
+  KVM_PV_ASYNC_CLEANUP_PREPARE. The resources that had been set aside
+  will be freed during the execution of this command. This PV command
+  should ideally be issued by userspace from a separate thread. If a
+  fatal signal is received (or the process terminates naturally), the
+  command will terminate immediately without completing, and the normal
+  KVM shutdown procedure will take care of cleaning up all remaining
+  protected VMs.
+
 
  4.126 KVM_X86_SET_MSR_FILTER
  ----------------------------
author maintainer
#7 lore raw

PATCH v13 3/6 KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE

In reply to Claudio Imbrenda

Add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE to signal that the KVM_PV_ASYNC_DISABLE and KVM_PV_ASYNC_DISABLE_PREPARE commands for the KVM_S390_PV_COMMAND ioctl are available.

---
 arch/s390/kvm/kvm-s390.c | 3 +++
 include/uapi/linux/kvm.h | 1 +
 2 files changed, 4 insertions(+)
arch/s390/kvm/kvm-s390.c +3 -0
index 96ec6ecb6b74..4a20a6be6601 100644
@@ -618,6 +618,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_S390_BPB:
 		r = test_facility(82);
 		break;
+	case KVM_CAP_S390_PROTECTED_ASYNC_DISABLE:
+		r = async_destroy && is_prot_virt_host();
+		break;
 	case KVM_CAP_S390_PROTECTED:
 		r = is_prot_virt_host();
 		break;
include/uapi/linux/kvm.h +1 -0
index 02602c5c1975..9afe0084b2c5 100644
@@ -1177,6 +1177,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_VM_DISABLE_NX_HUGE_PAGES 220
 #define KVM_CAP_S390_ZPCI_OP 221
 #define KVM_CAP_S390_CPU_TOPOLOGY 222
+#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 225
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
Signature · 1 line
2.37.1
author maintainer
#8 lore raw

PATCH v13 4/6 KVM: s390: pv: avoid export before import if possible

In reply to Claudio Imbrenda

If the appropriate UV feature bit is set, there is no need to perform an export before import.

---
 arch/s390/kernel/uv.c | 2 ++
 1 file changed, 2 insertions(+)
arch/s390/kernel/uv.c +2 -0
index f9810d2a267c..b455646c8d74 100644
@@ -255,6 +255,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,
  */
 static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
 {
+	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
+		return false;
 	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
 		return false;
 	return atomic_read(&mm->context.protected_count) > 1;
Signature · 1 line
2.37.1
maintainer Reviewed-by on v13 4/6
#9 lore raw

Re: [PATCH v13 4/6] KVM: s390: pv: avoid export before import if possible

Quoted 14 lines · Claudio Imbrenda

If the appropriate UV feature bit is set, there is no need to perform an export before import.

---
  arch/s390/kernel/uv.c | 2 ++
  1 file changed, 2 insertions(+)
arch/s390/kernel/uv.c +0 -0
index f9810d2a267c..b455646c8d74 100644

arch/s390/kernel/uv.c static int make_secure_pte(pte_t *ptep, unsigned long addr, · line 257

Quoted 3 lines
arch/s390/kernel/uv.c +0 -0
@@ -255,6 +255,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,
   */
  static bool should_export_before_import(struct uv_cb_header *uvcb, struct mm_struct *mm)
  {

The misc feature indicates that the UV will automatically transfer ownership from one protected VM to another when importing a shared page.

Other than that:

Quoted 5 lines
arch/s390/kernel/uv.c +2 -0
+	if (test_bit_inv(BIT_UV_FEAT_MISC, &uv_info.uv_feature_indications))
+		return false;
  	if (uvcb->cmd == UVC_CMD_UNPIN_PAGE_SHARED)
  		return false;
  	return atomic_read(&mm->context.protected_count) > 1;
author maintainer
#10 lore raw

PATCH v13 5/6 KVM: s390: pv: support for Destroy fast UVC

In reply to Claudio Imbrenda

Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.

When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.

---
 arch/s390/include/asm/uv.h | 10 +++++++
 arch/s390/kvm/pv.c         | 57 ++++++++++++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 8 deletions(-)
arch/s390/include/asm/uv.h +10 -0
index be3ef9dd6972..28a9ad57b6f1 100644
@@ -34,6 +34,7 @@
 #define UVC_CMD_INIT_UV			0x000f
 #define UVC_CMD_CREATE_SEC_CONF		0x0100
 #define UVC_CMD_DESTROY_SEC_CONF	0x0101
+#define UVC_CMD_DESTROY_SEC_CONF_FAST	0x0102
 #define UVC_CMD_CREATE_SEC_CPU		0x0120
 #define UVC_CMD_DESTROY_SEC_CPU		0x0121
 #define UVC_CMD_CONV_TO_SEC_STOR	0x0200
@@ -81,6 +82,7 @@ enum uv_cmds_inst {
 	BIT_UVC_CMD_UNSHARE_ALL = 20,
 	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
 	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
+	BIT_UVC_CMD_DESTROY_SEC_CONF_FAST = 23,
 	BIT_UVC_CMD_DUMP_INIT = 24,
 	BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE = 25,
 	BIT_UVC_CMD_DUMP_CPU = 26,
@@ -230,6 +232,14 @@ struct uv_cb_nodata {
 	u64 reserved20[4];
 } __packed __aligned(8);
 
+/* Destroy Configuration Fast */
+struct uv_cb_destroy_fast {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 handle;
+	u64 reserved20[5];
+} __packed __aligned(8);
+
 /* Set Shared Access */
 struct uv_cb_share {
 	struct uv_cb_header header;
arch/s390/kvm/pv.c +49 -8

100 lines not on this page · raw message

Signature · 1 line
2.37.1
Reviewed-by on v13 5/6
#11 lore raw

Re: [PATCH v13 5/6] KVM: s390: pv: support for Destroy fast UVC

Quoting Claudio Imbrenda (2022-08-10 14:56:24)

Quoted 7 lines

Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.

When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.

maintainer Reviewed-by on v13 5/6
#12 lore raw

Re: [PATCH v13 5/6] KVM: s390: pv: support for Destroy fast UVC

In reply to Claudio Imbrenda
Quoted 7 lines · Claudio Imbrenda

Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.

When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.

Quoted 36 lines
---
  arch/s390/include/asm/uv.h | 10 +++++++
  arch/s390/kvm/pv.c         | 57 ++++++++++++++++++++++++++++++++------
  2 files changed, 59 insertions(+), 8 deletions(-)
arch/s390/include/asm/uv.h +9 -0
index be3ef9dd6972..28a9ad57b6f1 100644
@@ -34,6 +34,7 @@
  #define UVC_CMD_INIT_UV			0x000f
  #define UVC_CMD_CREATE_SEC_CONF		0x0100
  #define UVC_CMD_DESTROY_SEC_CONF	0x0101
+#define UVC_CMD_DESTROY_SEC_CONF_FAST	0x0102
  #define UVC_CMD_CREATE_SEC_CPU		0x0120
  #define UVC_CMD_DESTROY_SEC_CPU		0x0121
  #define UVC_CMD_CONV_TO_SEC_STOR	0x0200
@@ -81,6 +82,7 @@ enum uv_cmds_inst {
  	BIT_UVC_CMD_UNSHARE_ALL = 20,
  	BIT_UVC_CMD_PIN_PAGE_SHARED = 21,
  	BIT_UVC_CMD_UNPIN_PAGE_SHARED = 22,
+	BIT_UVC_CMD_DESTROY_SEC_CONF_FAST = 23,
  	BIT_UVC_CMD_DUMP_INIT = 24,
  	BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE = 25,
  	BIT_UVC_CMD_DUMP_CPU = 26,
@@ -230,6 +232,14 @@ struct uv_cb_nodata {
  	u64 reserved20[4];
  } __packed __aligned(8);
 
+/* Destroy Configuration Fast */
+struct uv_cb_destroy_fast {
+	struct uv_cb_header header;
+	u64 reserved08[2];
+	u64 handle;
+	u64 reserved20[5];
+} __packed __aligned(8);
author maintainer
#13 lore raw

PATCH v13 6/6 KVM: s390: pv: module parameter to fence asynchronous destroy

In reply to Claudio Imbrenda

Add the module parameter "async_destroy", to allow the asynchronous destroy mechanism to be switched off. This might be useful for debugging purposes.

The parameter is enabled by default since the feature is opt-in anyway.

---
 arch/s390/kvm/kvm-s390.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +7 -1
index 4a20a6be6601..8c7af96c4546 100644
@@ -209,7 +209,13 @@ unsigned int diag9c_forwarding_hz;
 module_param(diag9c_forwarding_hz, uint, 0644);
 MODULE_PARM_DESC(diag9c_forwarding_hz, "Maximum diag9c forwarding per second, 0 to turn off");
 
-static int async_destroy;
+/*
+ * allow asynchronous deinit for protected guests; enable by default since
+ * the feature is opt-in anyway
+ */
+static int async_destroy = 1;
+module_param(async_destroy, int, 0444);
+MODULE_PARM_DESC(async_destroy, "Asynchronous destroy for protected guests");
 
 /*
  * For now we handle at most 16 double words as this is what the s390 base
Signature · 1 line
2.37.1