KVM: TDX: Decrease TDX VM shutdown time
Subsystems (5)
v2 cover KVM: TDX: Decrease TDX VM shutdown time
Hi
The version 1 RFC:
https://lore.kernel.org/all/[email protected]/
listed 3 options and implemented option 2. Sean replied with code for option 1, which tested out OK, so here it is plus a commit log.
It depends upon kvm_trylock_all_vcpus(kvm) which is the assumed result of Maxim's work-in-progress here:
https://lore.kernel.org/all/[email protected]/
Note it is assumed that kvm_trylock_all_vcpus(kvm) follows the return value semantics of mutex_trylock() i.e. 1 means locks have been successfully acquired, 0 means not succesful.
Sean Christopherson (1):
KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
Documentation/virt/kvm/x86/intel-tdx.rst | 16 ++++++++ arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 63 ++++++++++++++++++++++---------- 3 files changed, 61 insertions(+), 19 deletions(-)
Regards
Adrian
PATCH v2 1/1 KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
From: Sean Christopherson <[email protected]>
Add sub-ioctl KVM_TDX_TERMINATE_VM to release the HKID prior to shutdown, which enables more efficient reclaim of private memory.
Private memory is removed from MMU/TDP when guest_memfds are closed. If the HKID has not been released, the TDX VM is still in RUNNABLE state, so pages must be removed using "Dynamic Page Removal" procedure (refer TDX Module Base spec) which involves a number of steps:
Block further address translation
Exit each VCPU
Clear Secure EPT entry
Flush/write-back/invalidate relevant caches
However, when the HKID is released, the TDX VM moves to TD_TEARDOWN state where all TDX VM pages are effectively unmapped, so pages can be reclaimed directly.
Reclaiming TD Pages in TD_TEARDOWN State was seen to decrease the total reclaim time. For example:
VCPUs Size (GB) Before (secs) After (secs) 4 18 72 24 32 107 517 134 64 400 5539 467
[Adrian: wrote commit message, added KVM_TDX_TERMINATE_VM documentation,
and moved cpus_read_lock() inside kvm->lock for consistency as reported by lockdep]
--- Documentation/virt/kvm/x86/intel-tdx.rst | 16 ++++++ arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/tdx.c | 63 +++++++++++++++++------- 3 files changed, 61 insertions(+), 19 deletions(-)
Documentation/virt/kvm/x86/intel-tdx.rst +16 -0
@@ -38,6 +38,7 @@ ioctl with TDX specific sub-ioctl() commands. KVM_TDX_INIT_MEM_REGION, KVM_TDX_FINALIZE_VM, KVM_TDX_GET_CPUID, + KVM_TDX_TERMINATE_VM, KVM_TDX_CMD_NR_MAX, };
@@ -214,6 +215,21 @@ struct kvm_cpuid2. __u32 padding[3]; }; +KVM_TDX_TERMINATE_VM +------------------- +:Type: vm ioctl +:Returns: 0 on success, <0 on error + +Release Host Key ID (HKID) to allow more efficient reclaim of private memory. +After this, the TD is no longer in a runnable state. + +Using KVM_TDX_TERMINATE_VM is optional. + +- id: KVM_TDX_TERMINATE_VM +- flags: must be 0 +- data: must be 0 +- hw_error: must be 0 + KVM TDX creation flow ===================== In addition to the standard KVM flow, new TDX ioctls need to be called. The
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -939,6 +939,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_MEM_REGION ,KVM_TDX_FINALIZE_VM ,KVM_TDX_GET_CPUID , +KVM_TDX_TERMINATE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/vmx/tdx.c +44 -19
120 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
arch/x86/kvm/vmx/tdx.c static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · line 2801
Quoted 10 lines · Adrian Hunter
...
arch/x86/kvm/vmx/tdx.c +8
-0+static int tdx_terminate_vm (struct kvm *kvm) +{ +int r =0 ; + +guard (mutex)(&kvm->lock); +cpus_read_lock (); + +if (!kvm_trylock_all_vcpus (kvm)) {
Does this need to be a trylock variant? Is userspace expected to keep retrying this operation indefinitely?
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +13
-0+ r = -EBUSY ; + goto out; + } + +kvm_vm_dead (kvm); +kvm_unlock_all_vcpus (kvm); + +__tdx_release_hkid (kvm, true); +out: +cpus_read_unlock (); +return r; +} +
Sean Christopherson
maintainerRe: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
Quoted 14 lines · Vishal Annapurve
arch/x86/kvm/vmx/tdx.c static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · line 2801
Quoted 10 lines · Adrian Hunter
...
arch/x86/kvm/vmx/tdx.c +8
-0+static int tdx_terminate_vm (struct kvm *kvm) +{ +int r =0 ; + +guard (mutex)(&kvm->lock); +cpus_read_lock (); + +if (!kvm_trylock_all_vcpus (kvm)) {Does this need to be a trylock variant? Is userspace expected to keep retrying this operation indefinitely?
Userspace is expected to not be stupid, i.e. not be doing things with vCPUs when terminating the VM. This is already rather unpleasant, I'd rather not have to think hard about what could go wrong if KVM has to wait on all vCPU mutexes.
Re: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
Quoted 15 lines · Sean Christopherson
Quoted 14 lines · Vishal Annapurve
arch/x86/kvm/vmx/tdx.c static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · line 2801
Quoted 10 lines · Adrian Hunter
...
arch/x86/kvm/vmx/tdx.c +8
-0+static int tdx_terminate_vm (struct kvm *kvm) +{ +int r =0 ; + +guard (mutex)(&kvm->lock); +cpus_read_lock (); + +if (!kvm_trylock_all_vcpus (kvm)) {Does this need to be a trylock variant? Is userspace expected to keep retrying this operation indefinitely?
No issue was seen in testing with a QEMU hack with no retrying. Presumably if user space is not doing anything with the TDX VM at the same time, then there should not be contention.
KVM_TDX_TERMINATE_VM is optional so it is not necessary to wait indefinitely.
Quoted 3 lines
Userspace is expected to not be stupid, i.e. not be doing things with vCPUs when terminating the VM. This is already rather unpleasant, I'd rather not have to think hard about what could go wrong if KVM has to wait on all vCPU mutexes.
Sean Christopherson
maintainerRe: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
In reply to Adrian HunterQuoted 21 lines · Adrian Hunter
From: Sean Christopherson <[email protected]>
Add sub-ioctl
KVM_TDX_TERMINATE_VMto release the HKID prior to shutdown, which enables more efficient reclaim of private memory.Private memory is removed from MMU/TDP when
guest_memfdsare closed. If the HKID has not been released, the TDX VM is still in RUNNABLE state, so pages must be removed using "Dynamic Page Removal" procedure (refer TDX Module Base spec) which involves a number of steps:
Block further address translation
Exit each VCPU
Clear Secure EPT entry
Flush/write-back/invalidate relevant cachesHowever, when the HKID is released, the TDX VM moves to
TD_TEARDOWNstate where all TDX VM pages are effectively unmapped, so pages can be reclaimed directly.Reclaiming TD Pages in
TD_TEARDOWNState was seen to decrease the total reclaim time. For example:
Quoted 8 lines
VCPUs Size (GB) Before (secs) After (secs) 4 18 72 24 32 107 517 134 64 400 5539 467[Adrian: wrote commit message, added
KVM_TDX_TERMINATE_VMdocumentation,
and movedcpus_read_lock()insidekvm->lockfor consistency as reported by lockdep]
/facepalm
I over-thought this. We've had an long-standing battle with kvm_lock vs. cpus_read_lock(), but this is kvm->lock, not kvm_lock. /sigh
arch/x86/kvm/vmx/tdx.c static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · line 2798
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+static int tdx_terminate_vm (struct kvm *kvm) +{ +int r =0 ; + +guard (mutex)(&kvm->lock);
With kvm->lock taken outside cpus_read_lock(), just handle KVM_TDX_TERMINATE_VM in the switch statement, i.e. let tdx_vm_ioctl() deal with kvm->lock.
Quoted 32 lines
arch/x86/kvm/vmx/tdx.c +19
-0+cpus_read_lock (); + +if (!kvm_trylock_all_vcpus (kvm)) { + r = -EBUSY ; + goto out; + } + +kvm_vm_dead (kvm); +kvm_unlock_all_vcpus (kvm); + +__tdx_release_hkid (kvm, true); +out: +cpus_read_unlock (); +return r; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;@@ -2805,6 +2827,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)if (tdx_cmd.hw_error)return -EINVAL ; +if (tdx_cmd.id ==KVM_TDX_TERMINATE_VM ) +return tdx_terminate_vm (kvm); +mutex_lock (&kvm->lock);switch (tdx_cmd.id) {Signature · 1 line
2.43.0
Re: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
Quoted 36 lines · Sean Christopherson
Quoted 21 lines · Adrian Hunter
From: Sean Christopherson <[email protected]>
Add sub-ioctl
KVM_TDX_TERMINATE_VMto release the HKID prior to shutdown, which enables more efficient reclaim of private memory.Private memory is removed from MMU/TDP when
guest_memfdsare closed. If the HKID has not been released, the TDX VM is still in RUNNABLE state, so pages must be removed using "Dynamic Page Removal" procedure (refer TDX Module Base spec) which involves a number of steps:
Block further address translation
Exit each VCPU
Clear Secure EPT entry
Flush/write-back/invalidate relevant cachesHowever, when the HKID is released, the TDX VM moves to
TD_TEARDOWNstate where all TDX VM pages are effectively unmapped, so pages can be reclaimed directly.Reclaiming TD Pages in
TD_TEARDOWNState was seen to decrease the total reclaim time. For example:Quoted 8 lines
VCPUs Size (GB) Before (secs) After (secs) 4 18 72 24 32 107 517 134 64 400 5539 467[Adrian: wrote commit message, added
KVM_TDX_TERMINATE_VMdocumentation,
and movedcpus_read_lock()insidekvm->lockfor consistency as reported by lockdep]/facepalm
I over-thought this. We've had an long-standing battle with
kvm_lockvs.cpus_read_lock(), but this iskvm->lock, notkvm_lock. /sigh
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd) · line 2798
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+static int tdx_terminate_vm (struct kvm *kvm) +{ +int r =0 ; + +guard (mutex)(&kvm->lock);With
kvm->locktaken outsidecpus_read_lock(), just handleKVM_TDX_TERMINATE_VMin the switch statement, i.e. lettdx_vm_ioctl()deal withkvm->lock.
Ok, also cpus_read_lock() can go back where it was in __tdx_release_hkid().
But also in __tdx_release_hkid(), there is
if (KVM_BUG_ON(refcount_read(&kvm->users_count) && !terminate, kvm)) return;
However, __tdx_td_init() calls tdx_mmu_release_hkid() on the error path so that is not correct.
Quoted 33 lines
Quoted 33 lines
arch/x86/kvm/vmx/tdx.c +19
-0+cpus_read_lock (); + +if (!kvm_trylock_all_vcpus (kvm)) { + r = -EBUSY ; + goto out; + } + +kvm_vm_dead (kvm); +kvm_unlock_all_vcpus (kvm); + +__tdx_release_hkid (kvm, true); +out: +cpus_read_unlock (); +return r; +} +int tdx_vm_ioctl (struct kvm *kvm,void __user *argp) {struct kvm_tdx_cmd tdx_cmd;@@ -2805,6 +2827,9 @@ int tdx_vm_ioctl(struct kvm *kvm, void __user *argp)if (tdx_cmd.hw_error)return -EINVAL ; +if (tdx_cmd.id ==KVM_TDX_TERMINATE_VM ) +return tdx_terminate_vm (kvm); +mutex_lock (&kvm->lock);switch (tdx_cmd.id) {Signature · 1 line
2.43.0
Re: [PATCH V2 1/1] KVM: TDX: Add sub-ioctl KVM_TDX_TERMINATE_VM
Quoted 48 lines · Adrian Hunter
Quoted 36 lines · Sean Christopherson
Quoted 21 lines · Adrian Hunter
From: Sean Christopherson <[email protected]>
Add sub-ioctl
KVM_TDX_TERMINATE_VMto release the HKID prior to shutdown, which enables more efficient reclaim of private memory.Private memory is removed from MMU/TDP when
guest_memfdsare closed. If the HKID has not been released, the TDX VM is still in RUNNABLE state, so pages must be removed using "Dynamic Page Removal" procedure (refer TDX Module Base spec) which involves a number of steps:
Block further address translation
Exit each VCPU
Clear Secure EPT entry
Flush/write-back/invalidate relevant cachesHowever, when the HKID is released, the TDX VM moves to
TD_TEARDOWNstate where all TDX VM pages are effectively unmapped, so pages can be reclaimed directly.Reclaiming TD Pages in
TD_TEARDOWNState was seen to decrease the total reclaim time. For example:Quoted 8 lines
VCPUs Size (GB) Before (secs) After (secs) 4 18 72 24 32 107 517 134 64 400 5539 467[Adrian: wrote commit message, added
KVM_TDX_TERMINATE_VMdocumentation,
and movedcpus_read_lock()insidekvm->lockfor consistency as reported by lockdep]/facepalm
I over-thought this. We've had an long-standing battle with
kvm_lockvs.cpus_read_lock(), but this iskvm->lock, notkvm_lock. /sighQuoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+static int tdx_terminate_vm(struct kvm *kvm) +{ + int r = 0; + + guard(mutex)(&kvm->lock);With
kvm->locktaken outsidecpus_read_lock(), just handleKVM_TDX_TERMINATE_VMin the switch statement, i.e. lettdx_vm_ioctl()deal withkvm->lock.Ok, also
cpus_read_lock()can go back where it was in__tdx_release_hkid().
Quoted 8 lines
But also in
__tdx_release_hkid(), there isif (KVM_BUG_ON(refcount_read(&kvm->users_count) && !terminate, kvm)) return;However,
__tdx_td_init()callstdx_mmu_release_hkid()on the error path so that is not correct.
So it ends up like this:
Documentation/virt/kvm/x86/intel-tdx.rst +16 -0
@@ -38,6 +38,7 @@ ioctl with TDX specific sub-ioctl() commands. KVM_TDX_INIT_MEM_REGION, KVM_TDX_FINALIZE_VM, KVM_TDX_GET_CPUID, + KVM_TDX_TERMINATE_VM, KVM_TDX_CMD_NR_MAX, };
@@ -214,6 +215,21 @@ struct kvm_cpuid2. __u32 padding[3]; }; +KVM_TDX_TERMINATE_VM +------------------- +:Type: vm ioctl +:Returns: 0 on success, <0 on error + +Release Host Key ID (HKID) to allow more efficient reclaim of private memory. +After this, the TD is no longer in a runnable state. + +Using KVM_TDX_TERMINATE_VM is optional. + +- id: KVM_TDX_TERMINATE_VM +- flags: must be 0 +- data: must be 0 +- hw_error: must be 0 + KVM TDX creation flow ===================== In addition to the standard KVM flow, new TDX ioctls need to be called. The
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -939,6 +939,7 @@ enum kvm_tdx_cmd_id {KVM_TDX_INIT_MEM_REGION ,KVM_TDX_FINALIZE_VM ,KVM_TDX_GET_CPUID , +KVM_TDX_TERMINATE_VM , KVM_TDX_CMD_NR_MAX, };
arch/x86/kvm/vmx/tdx.c +26 -15
78 lines not on this page · raw message