KVM: s390: pv: implement lazy destroy for reboot
Subsystems (4)
Claudio Imbrenda
maintainerv13 cover KVM: s390: pv: implement lazy destroy for reboot
Previously, when a protected VM was rebooted or when it was shut down, its memory was made unprotected, and then the protected VM itself was destroyed. Looping over the whole address space can take some time, considering the overhead of the various Ultravisor Calls (UVCs). This means that a reboot or a shutdown would take a potentially long amount of time, depending on the amount of used memory.
This patchseries implements a deferred destroy mechanism for protected guests. When a protected guest is destroyed, its memory can be cleared in background, allowing the guest to restart or terminate significantly faster than before.
There are 2 possibilities when a protected VM is torn down:
* it still has an address space associated (reboot case)
* it does not have an address space anymore (shutdown case)
For the reboot case, two new commands are available for the KVM_S390_PV_COMMAND:
KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. In this case the userspace process should issue a normal KVM_PV_DISABLE
KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asychronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all leftovers.
The idea is that userspace should first issue the KVM_PV_ASYNC_CLEANUP_PREPARE command, and in case of success, create a new thread and issue KVM_PV_ASYNC_CLEANUP_PERFORM from there. This also allows for proper accounting of the CPU time needed for the asynchronous teardown.
This means that the same address space can have memory belonging to more than one protected guest, although only one will be running, the others will in fact not even have any CPUs.
The shutdown case should be dealt with in userspace (e.g. using clone(CLONE_VM)).
A module parameter is also provided to disable the new functionality, which is otherwise enabled by default. This should not be an issue since the new functionality is opt-in anyway. This is mainly thought to aid debugging.
v12->v13
* drop the patches that have been already merged
* rebase
Claudio Imbrenda (6):
KVM: s390: pv: asynchronous destroy for reboot
KVM: s390: pv: api documentation for asynchronous destroy
KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
KVM: s390: pv: avoid export before import if possible
KVM: s390: pv: support for Destroy fast UVC
KVM: s390: pv: module parameter to fence asynchronous destroy
Documentation/virt/kvm/api.rst | 30 ++- arch/s390/include/asm/kvm_host.h | 2 + arch/s390/include/asm/uv.h | 10 + arch/s390/kernel/uv.c | 2 + arch/s390/kvm/kvm-s390.c | 55 ++++- arch/s390/kvm/kvm-s390.h | 3 + arch/s390/kvm/pv.c | 331 ++++++++++++++++++++++++++++++- include/uapi/linux/kvm.h | 3 + 8 files changed, 416 insertions(+), 20 deletions(-)
Signature · 1 line
2.37.1
Claudio Imbrenda
maintainerPATCH v13 1/6 KVM: s390: pv: asynchronous destroy for reboot
Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.
This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.
This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.
The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.
Two new PV commands are added for the KVM_S390_PV_COMMAND ioctl:
KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.
KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all remaining protected VMs.
--- arch/s390/include/asm/kvm_host.h | 2 + arch/s390/kvm/kvm-s390.c | 46 ++++- arch/s390/kvm/kvm-s390.h | 3 + arch/s390/kvm/pv.c | 290 +++++++++++++++++++++++++++++-- include/uapi/linux/kvm.h | 2 + 5 files changed, 325 insertions(+), 18 deletions(-)
arch/s390/include/asm/kvm_host.h +2 -0
@@ -942,6 +942,8 @@ struct kvm_s390_pv {unsigned long stor_base;void *stor_var;bool dumping; +void *set_aside; +struct list_head need_cleanup;struct mmu_notifier mmu_notifier; };
arch/s390/kvm/kvm-s390.c +38 -8
98 lines not on this page · raw message
arch/s390/kvm/kvm-s390.h +3 -0
@@ -243,6 +243,9 @@ static inline u32 kvm_s390_get_gisa_desc(struct kvm *kvm)/* implemented in pv.c */ int kvm_s390_pv_destroy_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc);int kvm_s390_pv_create_cpu (struct kvm_vcpu *vcpu,u16 *rc,u16 *rrc); +int kvm_s390_pv_set_aside (struct kvm *kvm,u16 *rc,u16 *rrc); +int kvm_s390_pv_deinit_aside_vm (struct kvm *kvm,u16 *rc,u16 *rrc); +int kvm_s390_pv_deinit_cleanup_all (struct kvm *kvm,u16 *rc,u16 *rrc);int kvm_s390_pv_deinit_vm (struct kvm *kvm,u16 *rc,u16 *rrc);int kvm_s390_pv_init_vm (struct kvm *kvm,u16 *rc,u16 *rrc); int kvm_s390_pv_set_sec_parms(struct kvm *kvm, void *hdr, u64 length, u16 *rc,
arch/s390/kvm/pv.c +280 -10
315 lines not on this page · raw message
include/uapi/linux/kvm.h +2 -0
@@ -1739,6 +1739,8 @@ enum pv_cmd_id {KVM_PV_UNSHARE_ALL ,KVM_PV_INFO ,KVM_PV_DUMP , +KVM_PV_ASYNC_CLEANUP_PREPARE , + KVM_PV_ASYNC_CLEANUP_PERFORM, }; struct kvm_pv_cmd {
Signature · 1 line
2.37.1
Janosch Frank
maintainerRe: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot
Quoted 95 lines · Claudio Imbrenda
Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.
This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.
This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.
The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.
Two new PV commands are added for the
KVM_S390_PV_COMMANDioctl:
KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown. The current VM will then continue immediately as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.
KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing. The rest of the normal KVM teardown process will take care of properly cleaning up all remaining protected VMs.Signed-off-by Claudio Imbrenda <[email protected]>--- arch/s390/include/asm/kvm_host.h | 2 + arch/s390/kvm/kvm-s390.c | 46 ++++- arch/s390/kvm/kvm-s390.h | 3 + arch/s390/kvm/pv.c | 290 +++++++++++++++++++++++++++++-- include/uapi/linux/kvm.h | 2 + 5 files changed, 325 insertions(+), 18 deletions(-)arch/s390/include/asm/kvm_host.h +2
-0@@ -942,6 +942,8 @@ struct kvm_s390_pv {unsigned long stor_base;void *stor_var;bool dumping; +void *set_aside; +struct list_head need_cleanup;struct mmu_notifier mmu_notifier; };arch/s390/kvm/kvm-s390.c +17
-1@@ -209,6 +209,8 @@ unsigned int diag9c_forwarding_hz;module_param (diag9c_forwarding_hz, uint,0644 );MODULE_PARM_DESC (diag9c_forwarding_hz,"Maximum diag9c forwarding per second, 0 to turn off" ); +static int async_destroy; +/* * For now we handle at most 16 double words as this is what the s390 base * kernel handles and stores in the prefix page. If we ever need to go beyond @@ -2504,9 +2506,13 @@ static int kvm_s390_pv_dmp(struct kvm *kvm, struct kvm_pv_cmd *cmd,static int kvm_s390_handle_pv (struct kvm *kvm,struct kvm_pv_cmd *cmd) { +const bool needslock = (cmd->cmd !=KVM_PV_ASYNC_CLEANUP_PERFORM ); +void __user *argp = (void __user *)cmd->data;int r =0 ;u16 dummy; -void __user *argp = (void __user *)cmd->data; + +if (needslock) +mutex_lock (&kvm->lock);switch (cmd->cmd) {case KVM_PV_ENABLE : {@@ -2540,6 +2546,28 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)set_bit (IRQ_PEND_EXT_SERVICE , &kvm->arch.float_int.masked_irqs);break ; } +case KVM_PV_ASYNC_CLEANUP_PREPARE : + r = -EINVAL ; +if (!kvm_s390_pv_is_protected (kvm) || !async_destroy) +break ; + + r =kvm_s390_cpus_from_pv (kvm, &cmd->rc, &cmd->rrc); +/* +* If a CPU could not be destroyed, destroy VM will also fail. +* There is no point in trying to destroy it. Instead return +* the rc and rrc from the first CPU that failed destroying.
arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2568
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+*/ +if (r) +break ; + r =kvm_s390_pv_set_aside (kvm, &cmd->rc, &cmd->rrc); + +/* no need to block service interrupts any more */ +clear_bit (IRQ_PEND_EXT_SERVICE , &kvm->arch.float_int.masked_irqs); +break ; +case KVM_PV_ASYNC_CLEANUP_PERFORM : +/* This must not be called while holding kvm->lock */
Two things:
I know that we don't need to check async_destroy since it will find nothing to cleanup because the command above is fenced. But I'd still appreciate the same check here.
Consider adding this to the comment:
", this is asserted inside the function."
Quoted 25 lines
arch/s390/kvm/kvm-s390.c +6
-1+ r =kvm_s390_pv_deinit_aside_vm (kvm, &cmd->rc, &cmd->rrc); +break ;case KVM_PV_DISABLE : { r = -EINVAL ;if (!kvm_s390_pv_is_protected (kvm))@@ -2553,7 +2581,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)*/ if (r)break ; - r =kvm_s390_pv_deinit_vm (kvm, &cmd->rc, &cmd->rrc); + r =kvm_s390_pv_deinit_cleanup_all (kvm, &cmd->rc, &cmd->rrc);/* no need to block service interrupts any more */ clear_bit (IRQ_PEND_EXT_SERVICE , &kvm->arch.float_int.masked_irqs);@@ -2703,6 +2731,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)default : r = -ENOTTY ; } +if (needslock) +mutex_unlock (&kvm->lock); +return r; }
arch/s390/kvm/kvm-s390.c long kvm_arch_vm_ioctl(struct file *filp, · line 2941
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +1
-1@@ -2907,9 +2938,8 @@ long kvm_arch_vm_ioctl(struct file *filp, r = -EINVAL ;break ; } -mutex_lock (&kvm->lock); +/* must be called without kvm->lock */
...as it will acquire and release it by itself.
Quoted 28 lines
arch/s390/kvm/kvm-s390.c +4
-5r =kvm_s390_handle_pv (kvm, &args); -mutex_unlock (&kvm->lock);if (copy_to_user (argp, &args,sizeof (args))) { r = -EFAULT ;break ;@@ -3228,6 +3258,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)kvm_s390_vsie_init (kvm);if (use_gisa)kvm_s390_gisa_init (kvm); +INIT_LIST_HEAD (&kvm->arch.pv.need_cleanup); + kvm->arch.pv.set_aside =NULL ;KVM_EVENT (3 ,"vm 0x%pK created by pid %u" , kvm, current->pid);return 0 ;@@ -3272,11 +3304,9 @@ void kvm_arch_destroy_vm(struct kvm *kvm)/* * We are already at the end of life and kvm->lock is not taken. * This is ok as the file descriptor is closed by now and nobody -* can mess with the pv state. To avoid lockdep_assert_held from -* complaining we do not use kvm_s390_pv_is_protected. +* can mess with the pv state. */ -if (kvm_s390_pv_get_handle (kvm)) -kvm_s390_pv_deinit_vm (kvm, &rc, &rrc); +kvm_s390_pv_deinit_cleanup_all (kvm, &rc, &rrc);/* * Remove the mmu notifier only when the whole KVM VM is torn down, * and only if one was registered to begin with. If the VM is
[...]
Quoted 15 lines
arch/s390/kvm/kvm-s390.c +15
-0+ +/** +* kvm_s390_pv_set_aside - Set aside a protected VM for later teardown. +* @kvm: the VM +* @rc: return value for the RC field of the UVCB +* @rrc: return value for the RRC field of the UVCB +* +* Set aside the protected VM for a subsequent teardown. The VM will be able +* to continue immediately as a non-secure VM, and the information needed to +* properly tear down the protected VM is set aside. If another protected VM +* was already set aside without starting its teardown, this function will +* fail. +* The CPUs of the protected VM need to be destroyed beforehand. +* +* Context: kvm->lock needs to be held
arch/s390/kvm/kvm-s390.c
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +10
-0+* +* Return: 0 in case of success, -EINVAL if another protected VM was already set +* aside, -ENOMEM if the system ran out of memory. +*/ +int kvm_s390_pv_set_aside (struct kvm *kvm,u16 *rc,u16 *rrc) +{ +struct pv_vm_to_be_destroyed *priv; + +/* +* If another protected VM was already prepared, refuse.
s/prepared/set aside/
or
prepared for teardown
arch/s390/kvm/kvm-s390.c
Quoted 5 lines
arch/s390/kvm/kvm-s390.c +5
-0+* A normal deinitialization has to be performed instead. +*/ +if (kvm->arch.pv.set_aside) +return -EINVAL ; + priv =kmalloc (sizeof (*priv),GFP_KERNEL | __GFP_ZERO);
kzalloc()?
Quoted 8 lines
arch/s390/kvm/kvm-s390.c +8
-0+if (!priv) +return -ENOMEM ; + + priv->stor_var = kvm->arch.pv.stor_var; + priv->stor_base = kvm->arch.pv.stor_base; + priv->handle =kvm_s390_pv_get_handle (kvm); + priv->old_gmap_table = (unsigned long )kvm->arch.gmap->table; +WRITE_ONCE (kvm->arch.gmap->guest_handle,0 );
arch/s390/kvm/kvm-s390.c
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +8
-0+if (s390_replace_asce (kvm->arch.gmap)) { +kfree (priv); +return -ENOMEM ; } +kvm_s390_destroy_lower_2g (kvm); +kvm_s390_clear_pv_state (kvm); + kvm->arch.pv.set_aside = priv; + + *rc =1 ;
UVC_RC_EXECUTED
arch/s390/kvm/kvm-s390.c
Quoted 1 line
arch/s390/kvm/kvm-s390.c +1
-0+ *rrc =42 ;
I'd prefer setting the rrc to 0.
Quoted 2 lines
arch/s390/kvm/kvm-s390.c +2
-0+return 0 ; +}
Claudio Imbrenda
maintainerRe: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot
On Thu, 11 Aug 2022 18:26:13 +0200
Janosch Frank <[email protected]> wrote:
[...]
Quoted 17 lines
Quoted 17 lines
arch/s390/kvm/kvm-s390.c +17
-0+case KVM_PV_ASYNC_CLEANUP_PREPARE : + r = -EINVAL ; +if (!kvm_s390_pv_is_protected (kvm) || !async_destroy) +break ; + + r =kvm_s390_cpus_from_pv (kvm, &cmd->rc, &cmd->rrc); +/* +* If a CPU could not be destroyed, destroy VM will also fail. +* There is no point in trying to destroy it. Instead return +* the rc and rrc from the first CPU that failed destroying. +*/ +if (r) +break ; + r =kvm_s390_pv_set_aside (kvm, &cmd->rc, &cmd->rrc); + +/* no need to block service interrupts any more */ +clear_bit (IRQ_PEND_EXT_SERVICE , &kvm->arch.float_int.masked_irqs);
Quoted 8 lines
arch/s390/kvm/kvm-s390.c static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) · line 2568
Quoted 3 lines
arch/s390/kvm/kvm-s390.c +3
-0+break ; +case KVM_PV_ASYNC_CLEANUP_PERFORM : +/* This must not be called while holding kvm->lock */ Two things:
I know that we don't need to checkasync_destroysince it will find nothing to cleanup because the command above is fenced. But I'd still appreciate the same check here.
will add
Quoted 3 lines
Consider adding this to the comment:
", this is asserted inside the function."
will add
Quoted 25 lines
Quoted 25 lines
arch/s390/kvm/kvm-s390.c +6
-1+ r =kvm_s390_pv_deinit_aside_vm (kvm, &cmd->rc, &cmd->rrc); +break ;case KVM_PV_DISABLE : { r = -EINVAL ;if (!kvm_s390_pv_is_protected (kvm))@@ -2553,7 +2581,7 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)*/ if (r)break ; - r =kvm_s390_pv_deinit_vm (kvm, &cmd->rc, &cmd->rrc); + r =kvm_s390_pv_deinit_cleanup_all (kvm, &cmd->rc, &cmd->rrc);/* no need to block service interrupts any more */ clear_bit (IRQ_PEND_EXT_SERVICE , &kvm->arch.float_int.masked_irqs);@@ -2703,6 +2731,9 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd)default : r = -ENOTTY ; } +if (needslock) +mutex_unlock (&kvm->lock); +return r; }
Quoted 8 lines
arch/s390/kvm/kvm-s390.c long kvm_arch_vm_ioctl(struct file *filp, · line 2941
Quoted 6 lines
arch/s390/kvm/kvm-s390.c +1
-1@@ -2907,9 +2938,8 @@ long kvm_arch_vm_ioctl(struct file *filp, r = -EINVAL ;break ; } -mutex_lock (&kvm->lock); +/* must be called without kvm->lock */ ...as it will acquire and release it by itself.
none of the other switch cases acquire kvm->lock, I actually think the comment is redundant as it is, I don't think we need to expand it further.
Quoted 51 lines
Quoted 29 lines
arch/s390/kvm/kvm-s390.c +4
-5r =kvm_s390_handle_pv (kvm, &args); -mutex_unlock (&kvm->lock);if (copy_to_user (argp, &args,sizeof (args))) { r = -EFAULT ;break ;@@ -3228,6 +3258,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)kvm_s390_vsie_init (kvm);if (use_gisa)kvm_s390_gisa_init (kvm); +INIT_LIST_HEAD (&kvm->arch.pv.need_cleanup); + kvm->arch.pv.set_aside =NULL ;KVM_EVENT (3 ,"vm 0x%pK created by pid %u" , kvm, current->pid);return 0 ;@@ -3272,11 +3304,9 @@ void kvm_arch_destroy_vm(struct kvm *kvm)/* * We are already at the end of life and kvm->lock is not taken. * This is ok as the file descriptor is closed by now and nobody -* can mess with the pv state. To avoid lockdep_assert_held from -* complaining we do not use kvm_s390_pv_is_protected. +* can mess with the pv state. */ -if (kvm_s390_pv_get_handle (kvm)) -kvm_s390_pv_deinit_vm (kvm, &rc, &rrc); +kvm_s390_pv_deinit_cleanup_all (kvm, &rc, &rrc);/* * Remove the mmu notifier only when the whole KVM VM is torn down, * and only if one was registered to begin with. If the VM is [...]
Quoted 21 lines
arch/s390/kvm/pv.c +21
-0+ +/** +* kvm_s390_pv_set_aside - Set aside a protected VM for later teardown. +* @kvm: the VM +* @rc: return value for the RC field of the UVCB +* @rrc: return value for the RRC field of the UVCB +* +* Set aside the protected VM for a subsequent teardown. The VM will be able +* to continue immediately as a non-secure VM, and the information needed to +* properly tear down the protected VM is set aside. If another protected VM +* was already set aside without starting its teardown, this function will +* fail. +* The CPUs of the protected VM need to be destroyed beforehand. +* +* Context: kvm->lock needs to be held +* +* Return: 0 in case of success, -EINVAL if another protected VM was already set +* aside, -ENOMEM if the system ran out of memory. +*/ +int kvm_s390_pv_set_aside (struct kvm *kvm,u16 *rc,u16 *rrc) +{
Quoted 8 lines
arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 274
Quoted 4 lines
arch/s390/kvm/pv.c +4
-0+struct pv_vm_to_be_destroyed *priv; + +/* +* If another protected VM was already prepared, refuse. s/prepared/set aside/
or
prepared for teardown
prepared for teardown; will fix
Quoted 8 lines
arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 279
Quoted 6 lines
arch/s390/kvm/pv.c +5
-0+* A normal deinitialization has to be performed instead. +*/ +if (kvm->arch.pv.set_aside) +return -EINVAL ; + priv =kmalloc (sizeof (*priv),GFP_KERNEL | __GFP_ZERO);
kzalloc()?
oops, yes
Quoted 14 lines
Quoted 14 lines
arch/s390/kvm/pv.c +11
-0+if (!priv) +return -ENOMEM ; + + priv->stor_var = kvm->arch.pv.stor_var; + priv->stor_base = kvm->arch.pv.stor_base; + priv->handle =kvm_s390_pv_get_handle (kvm); + priv->old_gmap_table = (unsigned long )kvm->arch.gmap->table; +WRITE_ONCE (kvm->arch.gmap->guest_handle,0 ); +if (s390_replace_asce (kvm->arch.gmap)) { +kfree (priv); +return -ENOMEM ; }
Quoted 7 lines
arch/s390/kvm/pv.c static int kvm_s390_pv_alloc_vm(struct kvm *kvm) · line 297
Quoted 5 lines
arch/s390/kvm/pv.c +5
-0+kvm_s390_destroy_lower_2g (kvm); +kvm_s390_clear_pv_state (kvm); + kvm->arch.pv.set_aside = priv; + + *rc =1 ;
UVC_RC_EXECUTED
will fix
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +1
-0+ *rrc = 42;I'd prefer setting the rrc to 0.
I'd like to convey the information that the "successful" execution was actually faked
Quoted 3 lines
Quoted 3 lines
(quoted diff) +2
-0+ return 0; +}
Re: [PATCH v13 1/6] KVM: s390: pv: asynchronous destroy for reboot
In reply to Claudio ImbrendaQuoting Claudio Imbrenda (2022-08-10 14:56:20)
Quoted 12 lines
Until now, destroying a protected guest was an entirely synchronous operation that could potentially take a very long time, depending on the size of the guest, due to the time needed to clean up the address space from protected pages.
This patch implements an asynchronous destroy mechanism, that allows a protected guest to reboot significantly faster than previously.
This is achieved by clearing the pages of the old guest in background. In case of reboot, the new guest will be able to run in the same address space almost immediately.
Quoted 7 lines
The old protected guest is then only destroyed when all of its memory has been destroyed or otherwise made non protected.
Two new PV commands are added for the
KVM_S390_PV_COMMANDioctl:
KVM_PV_ASYNC_CLEANUP_PREPARE: prepares the current protected VM for asynchronous teardown.
I would add:
A protected VM which has been prepared for asynchronous teardown is called a *set aside VM*. A set a side VM never has any CPUs associated with it, but is still registered with the Ultravisor and may have memory.
Quoted 10 lines
The current VM will then continue immediately
as non-protected. If a protected VM had already been set aside without starting the teardown process, this call will fail. There can be at most one prepared VM at any time.
KVM_PV_ASYNC_CLEANUP_PERFORM: tears down the protected VM previously set aside for asynchronous teardown. This PV command should ideally be issued by userspace from a separate thread. If a fatal signal is received (or the process terminates naturally), the command will terminate immediately without completing.
I would add:
A set aside VM where cleanup started but was interrupted is called a *leftover VM*. There can be multiple leftovers per VM, which are tracked in the need_cleanup list.
I would like a more consistent language. We have three different names for the leftover in the patch below:
- leftover,
- the struct describing a leftover is called pv_vm_to_be_destroyed,
- the list of leftover vms is called need_cleanup.
It would be nice if this were a little more consistent, e.g. by renaming pv_vm_to_be_destroyed to pvm_vm_leftover and maybe need_cleanup to leftovers.
Otherwise, this looks good. I would be glad if you pick up my naming suggestions, but feel free to add:
Claudio Imbrenda
maintainerPATCH v13 2/6 KVM: s390: pv: api documentation for asynchronous destroy
In reply to Claudio ImbrendaAdd documentation for the new commands added to the KVM_S390_PV_COMMAND ioctl.
--- Documentation/virt/kvm/api.rst | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +28 -2
@@ -5163,8 +5163,11 @@ KVM_PV_ENABLE KVM_PV_DISABLE Deregister the VM from the Ultravisor and reclaim the memory that had been donated to the Ultravisor, making it usable by the kernel - again. All registered VCPUs are converted back to non-protected - ones. + again. All registered VCPUs are converted back to non-protected + ones. If a previous VM had been set aside for asynchonous teardown + with KVM_PV_ASYNC_CLEANUP_PREPARE and not actually torn down with + KVM_PV_ASYNC_CLEANUP_PERFORM, it will be torn down in this call + together with the current VM. KVM_PV_VM_SET_SEC_PARMS Pass the image header from VM memory to the Ultravisor in
@@ -5287,6 +5290,29 @@ KVM_PV_DUMP authentication tag all of which are needed to decrypt the dump at a later time. +KVM_PV_ASYNC_CLEANUP_PREPARE + Prepare the current protected VM for asynchronous teardown. Most + resources used by the current protected VM will be set aside for a + subsequent asynchronous teardown. The current protected VM will then + resume execution immediately as non-protected. There can be at most + one protected VM set aside at any time. If a protected VM had + already been set aside without starting the asynchronous teardown + process, this call will fail. In that case, the userspace process + should issue a normal KVM_PV_DISABLE. The resources set aside with + this call will need to be cleaned up with a subsequent call to + KVM_PV_ASYNC_CLEANUP_PERFORM or KVM_PV_DISABLE, otherwise they will + be cleaned up when KVM terminates. + +KVM_PV_ASYNC_CLEANUP_PERFORM + Tear down the protected VM previously set aside with + KVM_PV_ASYNC_CLEANUP_PREPARE. The resources that had been set aside + will be freed during the execution of this command. This PV command + should ideally be issued by userspace from a separate thread. If a + fatal signal is received (or the process terminates naturally), the + command will terminate immediately without completing, and the normal + KVM shutdown procedure will take care of cleaning up all remaining + protected VMs. + 4.126 KVM_X86_SET_MSR_FILTER ----------------------------
Signature · 1 line
2.37.1
Janosch Frank
maintainerRe: [PATCH v13 2/6] KVM: s390: pv: api documentation for asynchronous destroy
Quoted 14 lines · Claudio Imbrenda
Add documentation for the new commands added to the
KVM_S390_PV_COMMANDioctl.Signed-off-by Claudio Imbrenda <[email protected]>Reviewed-by Nico Boehr <[email protected]>--- Documentation/virt/kvm/api.rst | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-)Documentation/virt/kvm/api.rst +0
-0
Documentation/virt/kvm/api.rst KVM_PV_ENABLE · line 5168
Quoted 8 lines
Documentation/virt/kvm/api.rst +3
-2@@ -5163,8 +5163,11 @@ KVM_PV_ENABLE KVM_PV_DISABLE Deregister the VM from the Ultravisor and reclaim the memory that had been donated to the Ultravisor, making it usable by the kernel - again. All registered VCPUs are converted back to non-protected - ones. + again. All registered VCPUs are converted back to non-protected + ones. If a previous VM had been set aside for asynchonous teardown + with KVM_PV_ASYNC_CLEANUP_PREPARE and not actually torn down with
...and hasn't yet been torn down with...
Documentation/virt/kvm/api.rst KVM_PV_ENABLE · line 5170
Quoted 2 lines
Documentation/virt/kvm/api.rst +2
-0+ KVM_PV_ASYNC_CLEANUP_PERFORM, it will be torn down in this call + together with the current VM.
current PV VM? or protected VM
I know it's missing in the unchanged paragraph above too but such is life.
Quoted 5 lines
Documentation/virt/kvm/api.rst +0
-0KVM_PV_VM_SET_SEC_PARMS Pass the image header from VM memory to the Ultravisor in@@ -5287,6 +5290,29 @@ KVM_PV_DUMP authentication tag all of which are needed to decrypt the dump at a
Documentation/virt/kvm/api.rst KVM_PV_DUMP · line 5300
Quoted 10 lines
Documentation/virt/kvm/api.rst +8
-0later time. +KVM_PV_ASYNC_CLEANUP_PREPARE + Prepare the current protected VM for asynchronous teardown. Most + resources used by the current protected VM will be set aside for a + subsequent asynchronous teardown. The current protected VM will then + resume execution immediately as non-protected. There can be at most + one protected VM set aside at any time. If a protected VM had + already been set aside without starting the asynchronous teardown + process, this call will fail. In that case, the userspace process
If KVM_PV_ASYNC_CLEANUP_PREPARE has already been called without a successful KVM_PV_ASYNC_CLEANUP_PERFORM this call will fail. I.e. only be one PV VM can be set aside.
Do we need to finish the cleanup or is it enough to start the cleanup like you describe here?
Quoted 18 lines
Documentation/virt/kvm/api.rst +15
-0+ should issue a normal KVM_PV_DISABLE. The resources set aside with + this call will need to be cleaned up with a subsequent call to + KVM_PV_ASYNC_CLEANUP_PERFORM or KVM_PV_DISABLE, otherwise they will + be cleaned up when KVM terminates. + +KVM_PV_ASYNC_CLEANUP_PERFORM + Tear down the protected VM previously set aside with + KVM_PV_ASYNC_CLEANUP_PREPARE. The resources that had been set aside + will be freed during the execution of this command. This PV command + should ideally be issued by userspace from a separate thread. If a + fatal signal is received (or the process terminates naturally), the + command will terminate immediately without completing, and the normal + KVM shutdown procedure will take care of cleaning up all remaining + protected VMs. + 4.126 KVM_X86_SET_MSR_FILTER ----------------------------
Claudio Imbrenda
maintainerPATCH v13 3/6 KVM: s390: pv: add KVM_CAP_S390_PROTECTED_ASYNC_DISABLE
In reply to Claudio ImbrendaAdd KVM_CAP_S390_PROTECTED_ASYNC_DISABLE to signal that the KVM_PV_ASYNC_DISABLE and KVM_PV_ASYNC_DISABLE_PREPARE commands for the KVM_S390_PV_COMMAND ioctl are available.
--- arch/s390/kvm/kvm-s390.c | 3 +++ include/uapi/linux/kvm.h | 1 + 2 files changed, 4 insertions(+)
arch/s390/kvm/kvm-s390.c +3 -0
@@ -618,6 +618,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_S390_BPB : r =test_facility (82 );break ; +case KVM_CAP_S390_PROTECTED_ASYNC_DISABLE : + r = async_destroy &&is_prot_virt_host (); +break ;case KVM_CAP_S390_PROTECTED : r =is_prot_virt_host ();break ;
include/uapi/linux/kvm.h +1 -0
@@ -1177,6 +1177,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_VM_DISABLE_NX_HUGE_PAGES 220#define KVM_CAP_S390_ZPCI_OP 221#define KVM_CAP_S390_CPU_TOPOLOGY 222 +#define KVM_CAP_S390_PROTECTED_ASYNC_DISABLE 225#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.37.1
Claudio Imbrenda
maintainerPATCH v13 4/6 KVM: s390: pv: avoid export before import if possible
In reply to Claudio ImbrendaIf the appropriate UV feature bit is set, there is no need to perform an export before import.
--- arch/s390/kernel/uv.c | 2 ++ 1 file changed, 2 insertions(+)
arch/s390/kernel/uv.c +2 -0
@@ -255,6 +255,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,*/ static bool should_export_before_import (struct uv_cb_header *uvcb,struct mm_struct *mm) { +if (test_bit_inv (BIT_UV_FEAT_MISC , &uv_info.uv_feature_indications)) +return false;if (uvcb->cmd ==UVC_CMD_UNPIN_PAGE_SHARED )return false;return atomic_read (&mm->context.protected_count) >1 ;
Signature · 1 line
2.37.1
Re: [PATCH v13 4/6] KVM: s390: pv: avoid export before import if possible
Quoted 14 lines · Claudio Imbrenda
If the appropriate UV feature bit is set, there is no need to perform an export before import.
Signed-off-by Claudio Imbrenda <[email protected]>Reviewed-by Nico Boehr <[email protected]>--- arch/s390/kernel/uv.c | 2 ++ 1 file changed, 2 insertions(+)arch/s390/kernel/uv.c +0
-0
arch/s390/kernel/uv.c static int make_secure_pte(pte_t *ptep, unsigned long addr, · line 257
Quoted 3 lines
arch/s390/kernel/uv.c +0
-0@@ -255,6 +255,8 @@ static int make_secure_pte(pte_t *ptep, unsigned long addr,*/ static bool should_export_before_import (struct uv_cb_header *uvcb,struct mm_struct *mm) {
The misc feature indicates that the UV will automatically transfer ownership from one protected VM to another when importing a shared page.
Other than that:
Quoted 5 lines
arch/s390/kernel/uv.c +2
-0+if (test_bit_inv (BIT_UV_FEAT_MISC , &uv_info.uv_feature_indications)) +return false;if (uvcb->cmd ==UVC_CMD_UNPIN_PAGE_SHARED )return false;return atomic_read (&mm->context.protected_count) >1 ;
Claudio Imbrenda
maintainerPATCH v13 5/6 KVM: s390: pv: support for Destroy fast UVC
In reply to Claudio ImbrendaAdd support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.
When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.
--- arch/s390/include/asm/uv.h | 10 +++++++ arch/s390/kvm/pv.c | 57 ++++++++++++++++++++++++++++++++------ 2 files changed, 59 insertions(+), 8 deletions(-)
arch/s390/include/asm/uv.h +10 -0
@@ -34,6 +34,7 @@#define UVC_CMD_INIT_UV 0x000f#define UVC_CMD_CREATE_SEC_CONF 0x0100#define UVC_CMD_DESTROY_SEC_CONF 0x0101 +#define UVC_CMD_DESTROY_SEC_CONF_FAST 0x0102#define UVC_CMD_CREATE_SEC_CPU 0x0120#define UVC_CMD_DESTROY_SEC_CPU 0x0121#define UVC_CMD_CONV_TO_SEC_STOR 0x0200
@@ -81,6 +82,7 @@ enum uv_cmds_inst {BIT_UVC_CMD_UNSHARE_ALL =20 ,BIT_UVC_CMD_PIN_PAGE_SHARED =21 ,BIT_UVC_CMD_UNPIN_PAGE_SHARED =22 , +BIT_UVC_CMD_DESTROY_SEC_CONF_FAST =23 ,BIT_UVC_CMD_DUMP_INIT =24 ,BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE =25 , BIT_UVC_CMD_DUMP_CPU = 26,
@@ -230,6 +232,14 @@ struct uv_cb_nodata {u64 reserved20[4 ]; } __packed__aligned(8) ; +/* Destroy Configuration Fast */ +struct uv_cb_destroy_fast { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 handle; +u64 reserved20[5 ]; +} __packed__aligned(8) ; +/* Set Shared Access */ struct uv_cb_share {struct uv_cb_header header;
arch/s390/kvm/pv.c +49 -8
100 lines not on this page · raw message
Signature · 1 line
2.37.1
Re: [PATCH v13 5/6] KVM: s390: pv: support for Destroy fast UVC
Quoting Claudio Imbrenda (2022-08-10 14:56:24)
Quoted 7 lines
Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.
When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.
Signed-off-by Claudio Imbrenda <[email protected]>
Re: [PATCH v13 5/6] KVM: s390: pv: support for Destroy fast UVC
In reply to Claudio ImbrendaQuoted 7 lines · Claudio Imbrenda
Add support for the Destroy Secure Configuration Fast Ultravisor call, and take advantage of it for asynchronous destroy.
When supported, the protected guest is destroyed immediately using the new UVC, leaving only the memory to be cleaned up asynchronously.
Signed-off-by Claudio Imbrenda <[email protected]>
Quoted 36 lines
--- arch/s390/include/asm/uv.h | 10 +++++++ arch/s390/kvm/pv.c | 57 ++++++++++++++++++++++++++++++++------ 2 files changed, 59 insertions(+), 8 deletions(-)arch/s390/include/asm/uv.h +9
-0@@ -34,6 +34,7 @@#define UVC_CMD_INIT_UV 0x000f#define UVC_CMD_CREATE_SEC_CONF 0x0100#define UVC_CMD_DESTROY_SEC_CONF 0x0101 +#define UVC_CMD_DESTROY_SEC_CONF_FAST 0x0102#define UVC_CMD_CREATE_SEC_CPU 0x0120#define UVC_CMD_DESTROY_SEC_CPU 0x0121#define UVC_CMD_CONV_TO_SEC_STOR 0x0200@@ -81,6 +82,7 @@ enum uv_cmds_inst {BIT_UVC_CMD_UNSHARE_ALL =20 ,BIT_UVC_CMD_PIN_PAGE_SHARED =21 ,BIT_UVC_CMD_UNPIN_PAGE_SHARED =22 , +BIT_UVC_CMD_DESTROY_SEC_CONF_FAST =23 ,BIT_UVC_CMD_DUMP_INIT =24 ,BIT_UVC_CMD_DUMP_CONFIG_STOR_STATE =25 , BIT_UVC_CMD_DUMP_CPU = 26,@@ -230,6 +232,14 @@ struct uv_cb_nodata {u64 reserved20[4 ]; } __packed__aligned(8) ; +/* Destroy Configuration Fast */ +struct uv_cb_destroy_fast { +struct uv_cb_header header; +u64 reserved08[2 ]; +u64 handle; +u64 reserved20[5 ]; +} __packed__aligned(8) ;
Claudio Imbrenda
maintainerPATCH v13 6/6 KVM: s390: pv: module parameter to fence asynchronous destroy
In reply to Claudio ImbrendaAdd the module parameter "async_destroy", to allow the asynchronous destroy mechanism to be switched off. This might be useful for debugging purposes.
The parameter is enabled by default since the feature is opt-in anyway.
--- arch/s390/kvm/kvm-s390.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +7 -1
@@ -209,7 +209,13 @@ unsigned int diag9c_forwarding_hz;module_param (diag9c_forwarding_hz, uint,0644 );MODULE_PARM_DESC (diag9c_forwarding_hz,"Maximum diag9c forwarding per second, 0 to turn off" ); -static int async_destroy; +/* +* allow asynchronous deinit for protected guests; enable by default since +* the feature is opt-in anyway +*/ +static int async_destroy =1 ; +module_param (async_destroy, int,0444 ); +MODULE_PARM_DESC (async_destroy,"Asynchronous destroy for protected guests" );/* * For now we handle at most 16 double words as this is what the s390 base
Signature · 1 line
2.37.1