Add AMD SEV and SEV-ES intra host migration support
Subsystems (4)
v3 cover Add AMD SEV and SEV-ES intra host migration support
Intra host migration provides a low-cost mechanism for userspace VMM upgrades. It is an alternative to traditional (i.e., remote) live migration. Whereas remote migration handles move a guest to a new host, intra host migration only handles moving a guest to a new userspace VMM within a host. This can be used to update, rollback, change flags of the VMM, etc. The lower cost compared to live migration comes from the fact that the guest's memory does not need to be copied between processes. A handle to the guest memory simply gets passed to the new VMM, this could be done via using /dev/shm with share=on or similar feature.
The guest state can be transferred from an old VMM to a new VMM as follows:
1. Export guest state from KVM to the old user-space VMM via a getter user-space/kernel API 2. Transfer guest state from old VMM to new VMM via IPC communication 3. Import guest state into KVM from the new user-space VMM via a setter user-space/kernel API VMMs by exporting from KVM using getters, sending that data to the new VMM, then setting it again in KVM.
In the common case for intra host migration, we can rely on the normal ioctls for passing data from one VMM to the next. SEV, SEV-ES, and other confidential compute environments make most of this information opaque, and render KVM ioctls such as "KVM_GET_REGS" irrelevant. As a result, we need the ability to pass this opaque metadata from one VMM to the next. The easiest way to do this is to leave this data in the kernel, and transfer ownership of the metadata from one KVM VM (or vCPU) to the next. For example, we need to move the SEV enabled ASID, VMSAs, and GHCB metadata from one VMM to the next. In general, we need to be able to hand off any data that would be unsafe/impossible for the kernel to hand directly to userspace (and cannot be reproduced using data that can be handed safely to userspace).
During the intra host send operation the SEV required metadata, the guest's ASID is loaded into a kvm wide hashmap keyed by a value given by userspace. This allows the userspace VMM to pass the key to the target VMM. Then on intra host receive the target VMM can be loaded with the metadata from the hashmap.
v3:
* Fix memory leak found by dan.carpenter@
v2:
* Added marcorr@ reviewed by tag
* Renamed function introduced in 1/3
* Edited with seanjc@'s review comments
** Cleaned up WARN usage
** Userspace makes random token now
* Edited with brijesh.singh@'s review comments ** Checks for different LAUNCH_* states in send function
v1: https://lore.kernel.org/kvm/[email protected]/
Peter Gonda (3):
KVM, SEV: Refactor out function for unregistering encrypted regions KVM, SEV: Add support for SEV intra host migration
KVM, SEV: Add support for SEV-ES intra host migration
.../virt/kvm/amd-memory-encryption.rst | 43 ++ arch/x86/kvm/svm/sev.c | 396 +++++++++++++++++- arch/x86/kvm/svm/svm.h | 1 + include/uapi/linux/kvm.h | 12 + 4 files changed, 433 insertions(+), 19 deletions(-)
base-commit: 7caa04b36f20
Signature · 1 line
2.32.0.93.g670b81a890-goog
PATCH v3 1/3 KVM, SEV: Refactor out function for unregistering encrypted regions
Factor out helper function for freeing the encrypted region list.
--- arch/x86/kvm/svm/sev.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
arch/x86/kvm/svm/sev.c +17 -9
@@ -1775,11 +1775,25 @@ int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd)return ret; } +static void unregister_enc_regions (struct kvm *kvm, +struct list_head *mem_regions) +{ +struct enc_region *pos, *q; + +lockdep_assert_held (&kvm->lock); + +if (list_empty (mem_regions)) +return ; + +list_for_each_entry_safe (pos, q, mem_regions, list) { +__unregister_enc_region_locked (kvm, pos); +cond_resched (); + } +} +void sev_vm_destroy (struct kvm *kvm) {struct kvm_sev_info *sev = &to_kvm_svm (kvm)->sev_info; -struct list_head *head = &sev->regions_list; -struct list_head *pos, *q;if (!sev_guest (kvm))return ;
@@ -1803,13 +1817,7 @@ void sev_vm_destroy(struct kvm *kvm)* if userspace was terminated before unregistering the memory regions * then lets unpin all the registered memory. */ -if (!list_empty (head)) { -list_for_each_safe (pos, q, head) { -__unregister_enc_region_locked (kvm, - list_entry(pos, struct enc_region, list)); -cond_resched (); - } - } +unregister_enc_regions (kvm, &sev->regions_list);mutex_unlock (&kvm->lock);
Signature · 1 line
2.32.0.432.gabb21c7263-goog
Sean Christopherson
maintainerRe: [PATCH 1/3 V3] KVM, SEV: Refactor out function for unregistering encrypted regions
Prefer "KVM: SVM:" or "KVM: SEV:" in the shortlog, i.e. colon instead of comma after KVM.
Quoted 29 lines · Peter Gonda
Factor out helper function for freeing the encrypted region list.
Signed-off-by Peter Gonda <[email protected]>Reviewed-by Brijesh Singh <[email protected]>Reviewed-by Marc Orr <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Cc David Rientjes <[email protected]>Cc Dr. David Alan Gilbert <[email protected]>Cc Brijesh Singh <[email protected]>Cc Vitaly Kuznetsov <[email protected]>Cc Wanpeng Li <[email protected]>Cc Jim Mattson <[email protected]>Cc Joerg Roedel <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>--- arch/x86/kvm/svm/sev.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)arch/x86/kvm/svm/sev.c +0
-0
arch/x86/kvm/svm/sev.c int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd) · line 1779
Quoted 5 lines
arch/x86/kvm/svm/sev.c +2
-0@@ -1775,11 +1775,25 @@ int svm_vm_copy_asid_from(struct kvm *kvm, unsigned int source_fd)return ret; } +static void unregister_enc_regions (struct kvm *kvm, +struct list_head *mem_regions)
Indentation is wonky. There's an extra tab and an extra space.
Quoted 26 lines
arch/x86/kvm/svm/sev.c +14
-2+{ +struct enc_region *pos, *q; + +lockdep_assert_held (&kvm->lock); + +if (list_empty (mem_regions)) +return ; + +list_for_each_entry_safe (pos, q, mem_regions, list) { +__unregister_enc_region_locked (kvm, pos); +cond_resched (); + } +} +void sev_vm_destroy (struct kvm *kvm) {struct kvm_sev_info *sev = &to_kvm_svm (kvm)->sev_info; -struct list_head *head = &sev->regions_list; -struct list_head *pos, *q;if (!sev_guest (kvm))return ;@@ -1803,13 +1817,7 @@ void sev_vm_destroy(struct kvm *kvm)* if userspace was terminated before unregistering the memory regions * then lets unpin all the registered memory. */
arch/x86/kvm/svm/sev.c void sev_vm_destroy(struct kvm *kvm) · line 1822
Quoted 10 lines
arch/x86/kvm/svm/sev.c +1
-7-if (!list_empty (head)) { -list_for_each_safe (pos, q, head) { -__unregister_enc_region_locked (kvm, - list_entry(pos, struct enc_region, list)); -cond_resched (); - } - } +unregister_enc_regions (kvm, &sev->regions_list);mutex_unlock (&kvm->lock);
Is there any reason for taking kvm->lock in this path? The VM is being destroyed, there should be no other references, i.e. this is the only task that can be doing anything with @kvm.
The lock is harmless, it just always gives me pause to see the cond_resched() while holding kvm->lock.
Quoted 3 lines
Signature · 1 line
2.32.0.432.gabb21c7263-goog
PATCH v3 2/3 KVM, SEV: Add support for SEV intra host migration
In reply to Peter GondaFor SEV to work with intra host migration, contents of the SEV info struct such as the ASID (used to index the encryption key in the AMD SP) and the list of memory regions need to be transferred to the target VM. This change adds commands for sending and receiving the sev info.
To avoid exposing this internal state to userspace and prevent other processes from importing state they shouldn't have access to, the send returns a token to userspace that is handed off to the target VM. The target passes in this token to receive the sent state. The token is only valid for one-time use. Functionality on the source becomes limited after send has been performed. If the source is destroyed before the target has received, the token becomes invalid.
The target is expected to be initialized (sev_guest_init), but not launched state (sev_launch_start) when performing receive. Once the target has received, it will be in a launched state and will not need to perform the typical SEV launch commands.
--- .../virt/kvm/amd-memory-encryption.rst | 43 +++ arch/x86/kvm/svm/sev.c | 248 +++++++++++++++++- arch/x86/kvm/svm/svm.h | 1 + include/uapi/linux/kvm.h | 12 + 4 files changed, 294 insertions(+), 10 deletions(-)
Documentation/virt/kvm/amd-memory-encryption.rst +43 -0
@@ -427,6 +427,49 @@ issued by the hypervisor to make the guest ready for execution. Returns: 0 on success, -negative on error +10. KVM_SEV_INTRA_HOST_SEND +---------------------------------- + +The KVM_SEV_INTRA_HOST_SEND command is used to stage the VM's SEV info +for the purposes of migrating memory to a new local VM while using the same SEV +key. If the source VM is destroyed before the staged info has been received by +the target, the info is lost. Once the info has been staged, only commands +KVM_SEV_DBG_DECRYPT, and KVM_SEV_DBG_ENCRYPT +can be used by the source. + +Parameters (in): struct kvm_sev_intra_host_send + +Returns: 0 on success, -negative on error + +:: + + struct kvm_sev_intra_host_send { + __u64 info_token; /* token referencing the staged info */ + }; + +11. KVM_SEV_INTRA_HOST_RECEIVE +------------------------------------- + +The KVM_SEV_INTRA_HOST_RECEIVE command is used to transfer staged SEV +info to a target VM from some source VM. SEV on the target VM should be active +when receive is performed, but not yet launched and without any pinned memory. +The launch commands should be skipped after receive because they should have +already been performed on the source. + +Parameters (in/out): struct kvm_sev_intra_host_receive + +Returns: 0 on success, -negative on error + +:: + + struct kvm_sev_intra_host_receive { + __u64 info_token; /* token referencing the staged info */ + __u32 handle; /* guest handle */ + }; + +On success, the 'handle' field contains the handle for this SEV guest. + + References ==========
arch/x86/kvm/svm/sev.c +238 -10
311 lines not on this page · raw message
arch/x86/kvm/svm/svm.h +1 -0
@@ -79,6 +79,7 @@ struct kvm_sev_info {u64 ap_jump_table;/* SEV-ES AP Jump Table address */ struct kvm *enc_context_owner;/* Owner of copied encryption context */ struct misc_cg *misc_cg;/* For misc cgroup accounting */ +u64 info_token;/* Token for SEV info intra host migration */ };struct kvm_svm {
include/uapi/linux/kvm.h +12 -0
@@ -1709,6 +1709,9 @@ enum sev_cmd_id {KVM_SEV_GET_ATTESTATION_REPORT ,/* Guest Migration Extension */ KVM_SEV_SEND_CANCEL , +/* Intra host migration commands */ +KVM_SEV_INTRA_HOST_SEND , +KVM_SEV_INTRA_HOST_RECEIVE , KVM_SEV_NR_MAX, };
@@ -1806,6 +1809,15 @@ struct kvm_sev_receive_update_data {__u32 trans_len; }; +struct kvm_sev_intra_host_send { +__u64 info_token; +}; + +struct kvm_sev_intra_host_receive { +__u64 info_token; +__u32 handle; +}; +#define KVM_DEV_ASSIGN_ENABLE_IOMMU (1 << 0)#define KVM_DEV_ASSIGN_PCI_2_3 (1 << 1)#define KVM_DEV_ASSIGN_MASK_INTX (1 << 2)
Signature · 1 line
2.32.0.432.gabb21c7263-goog
Sean Christopherson
maintainerRe: [PATCH 2/3 V3] KVM, SEV: Add support for SEV intra host migration
Quoted 7 lines · Peter Gonda
To avoid exposing this internal state to userspace and prevent other processes from importing state they shouldn't have access to, the send returns a token to userspace that is handed off to the target VM. The target passes in this token to receive the sent state. The token is only valid for one-time use. Functionality on the source becomes limited after send has been performed. If the source is destroyed before the target has received, the token becomes invalid.
...
Quoted 7 lines
+11.
KVM_SEV_INTRA_HOST_RECEIVE
+-------------------------------------Documentation/virt/kvm/amd-memory-encryption.rst +5
-0+ +The KVM_SEV_INTRA_HOST_RECEIVE command is used to transfer staged SEV +info to a target VM from some source VM. SEV on the target VM should be active +when receive is performed, but not yet launched and without any pinned memory. +The launch commands should be skipped after receive because they should have
Documentation/virt/kvm/amd-memory-encryption.rst issued by the hypervisor to make the guest ready for execution. · line 466
Quoted 10 lines
Documentation/virt/kvm/amd-memory-encryption.rst +10
-0+already been performed on the source. + +Parameters (in/out): struct kvm_sev_intra_host_receive + +Returns: 0 on success, -negative on error + +:: + + struct kvm_sev_intra_host_receive { + __u64 info_token; /* token referencing the staged info */
Sorry to belatedly throw a wrench in things, but why use a token approach? This is only intended for migrating between two userspace VMMs using the same KVM module, which can access both the source and target KVM instances (VMs/guests). Rather than indirectly communicate through a token, why not communidate directly? Same idea as svm_vm_copy_asid_from().
The locking needs special consideration, e.g. attempting to take kvm->lock on both the source and dest could deadlock if userspace is malicious and double-migrates, but I think a flag and global spinlock to state that migration is in-progress would suffice.
Locking aside, this would reduce the ABI to a single ioctl(), should avoid most if not all temporary memory allocations, and would obviate the need for patch 1 since there's no limbo state, i.e. the encrypted regions are either owned by the source or the dest.
I think the following would work? Another thought would be to make the helpers and "lock for multi-lock" flag arch-agnostic, e.g. the logic below works iff this is the only path that takes two kvm->locks simultaneous.
static int svm_sev_lock_for_migration(struct kvm *kvm)
{
struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
int ret = 0;
/*
* Bail if this VM is already involved in a migration to avoid deadlock
* between two VMs trying to migrate to/from each other.
*/
spin_lock(&sev_migration_lock);
if (sev->migration_in_progress)
ret = -EINVAL;
else
sev->migration_in_progress = true;
spin_unlock(&sev_migration_lock);
if (!ret)
mutex_lock(&kvm->lock);
return ret;
}static void svm_unlock_after_migration(struct kvm *kvm)
{
mutex_unlock(&kvm->lock);
WRITE_ONCE(sev->migration_in_progress, false);
}int svm_sev_migrate_from(struct kvm *kvm, unsigned int source_fd)
{
struct file *source_kvm_file;
struct kvm *source_kvm;
int ret = -EINVAL;
ret = svm_sev_lock_for_migration(kvm);
if (ret)
return ret;
if (!sev_guest(kvm))
goto out_unlock;
source_kvm_file = fget(source_fd);
if (!file_is_kvm(source_kvm_file)) {
ret = -EBADF;
goto out_fput;
}
source_kvm = source_kvm_file->private_data;
ret = svm_sev_lock_for_migration(source_kvm);
if (ret)
goto out_fput;
if (!sev_guest(source_kvm)) {
ret = -EINVAL;
goto out_source;
}
<migration magic>out_source:
svm_unlock_after_migration(&source_kvm->lock); out_fpu: if (source_kvm_file) fput(source_kvm_file); out_unlock: svm_unlock_after_migration(kvm); return ret; }
PATCH v3 3/3 KVM, SEV: Add support for SEV-ES intra host migration
In reply to Peter GondaFor SEV-ES to work with intra host migration the VMSAs, GHCB metadata, and other SEV-ES info needs to be preserved along with the guest's memory.
--- arch/x86/kvm/svm/sev.c | 145 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 135 insertions(+), 10 deletions(-)
arch/x86/kvm/svm/sev.c +135 -10
@@ -77,6 +77,19 @@ struct enc_region {unsigned long size; }; +struct vmsa_node { +struct list_head list; +int vcpu_id; +struct vmcb_save_area *vmsa; +struct ghcb *ghcb; +u64 ghcb_gpa; + +void *ghcb_sa; +u64 ghcb_sa_len; +bool ghcb_sa_sync; +bool ghcb_sa_free; +}; +struct sev_info_migration_node {struct hlist_node hnode;u64 token;
@@ -87,6 +100,11 @@ struct sev_info_migration_node {unsigned long pages_locked;struct list_head regions_list;struct misc_cg *misc_cg; + +/* The following fields are for SEV-ES guests */ +bool es_enabled; +struct list_head vmsa_list; +u64 ap_jump_table; };#define SEV_INFO_MIGRATION_HASH_BITS 7
@@ -1157,6 +1175,87 @@ static int place_migration_node(struct sev_info_migration_node *entry)return ret; } +static void process_vmsa_list (struct kvm *kvm,struct list_head *vmsa_list) +{ +struct vmsa_node *vmsa_node, *q; +struct kvm_vcpu *vcpu; +struct vcpu_svm *svm; + +lockdep_assert_held (&kvm->lock); + +/* +* Move any stashed VMSAs back to their respective VMCBs and delete +* those nodes. +*/ +list_for_each_entry_safe (vmsa_node, q, vmsa_list, list) { + vcpu =kvm_get_vcpu_by_id (kvm, vmsa_node->vcpu_id); +if (WARN_ON (!vcpu)) +continue ; + + svm =to_svm (vcpu); + svm->vmsa = vmsa_node->vmsa; + svm->ghcb = vmsa_node->ghcb; + svm->vmcb->control.ghcb_gpa = vmsa_node->ghcb_gpa; + svm->vcpu.arch.guest_state_protected = true; + svm->vmcb->control.vmsa_pa =__pa (svm->vmsa); + svm->ghcb_sa = vmsa_node->ghcb_sa; + svm->ghcb_sa_len = vmsa_node->ghcb_sa_len; + svm->ghcb_sa_sync = vmsa_node->ghcb_sa_sync; + svm->ghcb_sa_free = vmsa_node->ghcb_sa_free; + +list_del (&vmsa_node->list); +kfree (vmsa_node); + } +} + +static int create_vmsa_list (struct kvm *kvm, +struct sev_info_migration_node *entry) +{ +int i; +const int num_vcpus =atomic_read (&kvm->online_vcpus); +struct vmsa_node *node; +struct kvm_vcpu *vcpu; +struct vcpu_svm *svm; + + +INIT_LIST_HEAD (&entry->vmsa_list); +for (i =0 ; i < num_vcpus; ++i) { + node =kzalloc (sizeof (*node),GFP_KERNEL ); +if (!node) + goto e_freelist; + + vcpu = kvm->vcpus[i]; +if (!vcpu->arch.guest_state_protected) { +kfree (node); + goto e_freelist; + } + + node->vcpu_id = vcpu->vcpu_id; + svm =to_svm (vcpu); + node->vmsa = svm->vmsa; + svm->vmsa =NULL ; + node->ghcb = svm->ghcb; + svm->ghcb =NULL ; + node->ghcb_gpa = svm->vmcb->control.ghcb_gpa; + node->ghcb_sa = svm->ghcb_sa; + svm->ghcb_sa =NULL ; + node->ghcb_sa_len = svm->ghcb_sa_len; + svm->ghcb_sa_len =0 ; + node->ghcb_sa_sync = svm->ghcb_sa_sync; + svm->ghcb_sa_sync = false; + node->ghcb_sa_free = svm->ghcb_sa_free; + svm->ghcb_sa_free = false; + +list_add_tail (&node->list, &entry->vmsa_list); + } + +return 0 ; + +e_freelist: +process_vmsa_list (kvm, &entry->vmsa_list); +return -1 ; +} +static int sev_intra_host_send (struct kvm *kvm,struct kvm_sev_cmd *argp) {struct kvm_sev_info *sev = &to_kvm_svm (kvm)->sev_info;
@@ -1167,9 +1266,6 @@ static int sev_intra_host_send(struct kvm *kvm, struct kvm_sev_cmd *argp)if (!sev_guest (kvm))return -ENOTTY ; -if (sev->es_active) -return -EPERM ; -if (sev->handle ==0 )return -EPERM ;
@@ -1196,13 +1292,28 @@ static int sev_intra_host_send(struct kvm *kvm, struct kvm_sev_cmd *argp)INIT_LIST_HEAD (&entry->regions_list);list_replace_init (&sev->regions_list, &entry->regions_list); +if (sev_es_guest (kvm)) { +/* +* If this is an ES guest, we need to move each VMCB's VMSA +* into a list for migration. +*/ + entry->es_enabled = true; + entry->ap_jump_table = sev->ap_jump_table; +if (create_vmsa_list (kvm, entry)) + goto e_listdel; + } +if (place_migration_node (entry)) - goto e_listdel; + goto e_vmsadel; sev->info_token = entry->token;return 0 ; +e_vmsadel: +if (sev_es_guest (kvm)) +process_vmsa_list (kvm, &entry->vmsa_list); + e_listdel:list_replace_init (&entry->regions_list, &sev->regions_list);
@@ -1223,9 +1334,6 @@ static int sev_intra_host_receive(struct kvm *kvm,if (!sev_guest (kvm))return -ENOTTY ; -if (sev->es_active) -return -EPERM ; -if (sev->handle !=0 )return -EPERM ;
@@ -1242,6 +1350,13 @@ static int sev_intra_host_receive(struct kvm *kvm,memcpy (&old_info, sev,sizeof (old_info)); +if (entry->es_enabled) { +process_vmsa_list (kvm, &entry->vmsa_list); + + sev->es_active = true; + sev->ap_jump_table = entry->ap_jump_table; + } +/* * The source VM always frees @entry On the target we simply * mark the token as invalid to notify the source the sev info
@@ -2034,8 +2149,16 @@ void sev_vm_destroy(struct kvm *kvm)WARN_ON (sev->info_token && !list_empty (&sev->regions_list));unregister_enc_regions (kvm, &sev->regions_list); -if (mig_entry) +if (mig_entry) { +/* +* If there are any saved VMSAs, restore them so they can be +* destructed through the normal path. +*/ +if (mig_entry->es_enabled) +process_vmsa_list (kvm, &mig_entry->vmsa_list); +unregister_enc_regions (kvm, &mig_entry->regions_list); + }mutex_unlock (&kvm->lock);
@@ -2227,9 +2350,11 @@ void sev_free_vcpu(struct kvm_vcpu *vcpu) svm =to_svm (vcpu); -if (vcpu->arch.guest_state_protected) +if (svm->ghcb && vcpu->arch.guest_state_protected)sev_flush_guest_memory (svm, svm->vmsa,PAGE_SIZE ); -__free_page (virt_to_page (svm->vmsa)); + +if (svm->vmsa) +__free_page (virt_to_page (svm->vmsa));if (svm->ghcb_sa_free)kfree (svm->ghcb_sa);
Signature · 1 line
2.32.0.432.gabb21c7263-goog