Confidential guest-assisted live migration
v2 cover Confidential guest-assisted live migration
This is an RFC series for fast migration of confidential guests using an in-guest migration helper that lives in OVMF. QEMU VM live migration needs to read source VM's RAM and write it in the target VM; this mechanism doesn't work when the guest memory is encrypted or QEMU is prevented from reading it in another way. In order to support live migration in such scenarios, we introduce an in-guest migration helper which can securely extract RAM content from the guest in order to send it to the target. The migration helper is implemented as part of the VM's firmware in OVMF.
We've implemented and tested this on AMD SEV, but expect most of the processes can be used with other technologies that prevent direct access of hypervisor to the guest's memory. Specifically, we don't use SEV's PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all; but note that the mirror VM relies on KVM_CAP_VM_COPY_ENC_CONTEXT_FROM to shared the SEV ASID with the main VM.
Corresponding RFC patches for OVMF have been posted by Tobin Feldman-Fitzthum on edk2-devel [1]. Those include the crux of the migration helper: a mailbox protocol over a shared memory page which allows communication between QEMU and the migration helper. In the source VM this is used to read a page and encrypt it for transport; in the target it is used to decrypt the incoming page and storing the content in the correct address in the guest memory. All encryption and decryption operations occur inside the trusted context in the VM, and therefore the VM's memory plaintext content is never accessible to the hosts participating in the migration.
In order to allow OVMF to run the migration helper in parallel to the guest OS, we use a mirror VM [3], which shares the same memory mapping and SEV ASID as the main VM but has its own run loop. To start the mirror vcpu and the migration handler, we added a temporary start-migration-handler QMP command; this will be removed in a future version to run as part of the migrate QMP command.
In the target VM we need the migration handler running to receive incoming RAM pages; to achieve that, we boot the VM into OVMF with a special fw_cfg value that causes OVMF to not boot the guest OS; we then allow QEMU to receive an incoming migration by issuing a new start-migrate-incoming QMP command.
The confidential RAM migration requires checking whether a given guest RAM page is encrypted or not. This is achieved using SEV shared regions list tracking, which is implemented as part the SEV live migration patch series [2]. This feature tracks hypercalls from OVMF and guest Linux to report changes of page encryption status so that QEMU has an up-to-date view of which memory regions are shared and which are encrypted.
We left a few unfinished edges in this RFC but decided to publish it to start the commmunity discussion. TODOs:
1. QMP commands start-migration-handler and start-migrate-incoming are developer tools and should be performed automatically.
2. The entry point address of the in-guest migration handler and its GDT are currently hard-coded in QEMU (patch 8); instead they should be discovered using pc_system_ovmf_table_find. Same applies for the mailbox address (patch 1).
3. For simplicity, this patch series forces the use of the guest-assisted migration instead of the SEV PSP-based migration. Ideally we might want the user to choose the desired mode using migrate-set-parameters or a similar mechanism.
4. There is currently no discovery protocol between QEMU and OVMF to verify that OVMF indeed supports in-guest migration handler.
List of patches in this series:
1-3: introduce new confidtial RAM migration functions which communicate
with the migration helper.
4-6: use the new MH communication functions when migrating encrypted RAM
pages
7-9: allow starting migration handler on mirror vcpu with QMP command
start-migration-handler
10: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
11: fix devices issues when loading state into a live VM 12: add documentation
This patch series is based on top of:
1. Add SEV guest live migration support, from Ashish Kalra [2]
2. Support for mirror VM, from Ashish Kalra [3]
[1] https://edk2.groups.io/g/devel/message/79517
[2] https://lore.kernel.org/qemu-devel/[email protected]/
[3] https://lore.kernel.org/qemu-devel/[email protected]/
Changes from RFC v1:
- Use the an SEV mirror VM for the migation handler (instead of auxilliary vcpus)
RFC v1:
https://lore.kernel.org/qemu-devel/[email protected]/
Dov Murik (12):
migration: Add helpers to save confidential RAM
migration: Add helpers to load confidential RAM
migration: Introduce gpa_inside_migration_helper_shared_area
migration: Save confidential guest RAM using migration helper
migration: Load confidential guest RAM using migration helper
migration: Skip ROM, non-RAM, and vga.vram memory region during RAM
migration
i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc
migration: Allow resetting the mirror vcpu to the MH entry point
migration: Add QMP command start-migration-handler
migration: Add start-migrate-incoming QMP command hw/isa/lpc_ich9: Allow updating an already-running VM
docs: Add confidential guest live migration documentation
docs/confidential-guest-live-migration.rst | 145 +++++++++ docs/confidential-guest-support.txt | 5 + docs/index.rst | 1 + qapi/migration.json | 38 +++ include/sysemu/sev.h | 1 + migration/confidential-ram.h | 23 ++ hw/isa/lpc_ich9.c | 3 +- migration/confidential-ram.c | 339 +++++++++++++++++++++ migration/migration.c | 29 ++ migration/ram.c | 133 +++++++- target/i386/kvm/kvm.c | 4 +- migration/meson.build | 2 +- migration/trace-events | 4 + 13 files changed, 714 insertions(+), 13 deletions(-) create mode 100644 docs/confidential-guest-live-migration.rst create mode 100644 migration/confidential-ram.h create mode 100644 migration/confidential-ram.c
Signature · 1 line
2.20.1
RFC PATCH v2 1/12 migration: Add helpers to save confidential RAM
QEMU cannot read the memory of memory-encrypted guests, which is required for sending RAM to the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to extract pages from memory; these pages are encrypted with a transfer key that is known to the source and target guests, but not to both QEMUs.
The interaction with the guest migration helper is performed using two shared (unencrypted) pages which both QEMU and guest can read from and write to. The details of the mailbox protocol are described in migration/confidential-ram.c.
--- migration/confidential-ram.h | 17 ++++ migration/confidential-ram.c | 184 +++++++++++++++++++++++++++++++++++ migration/meson.build | 2 +- migration/trace-events | 3 + 4 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 migration/confidential-ram.h create mode 100644 migration/confidential-ram.c
migration/confidential-ram.h new file +17 -0
@@ -0,0 +1,17 @@ +/* +* QEMU migration for confidential guest's RAM +*/ + +#ifndef QEMU_CONFIDENTIAL_RAM_H +#define QEMU_CONFIDENTIAL_RAM_H + +#include "exec/cpu-common.h" +#include "qemu-file.h" + +void cgs_mh_init (void ); +void cgs_mh_cleanup (void ); + +int cgs_mh_save_encrypted_page (QEMUFile *f,ram_addr_t src_gpa,uint32_t size, +uint64_t *bytes_sent); + +#endif
migration/confidential-ram.c new file +184 -0
184 lines not on this page · raw message
migration/meson.build +1 -1
@@ -32,4 +32,4 @@ softmmu_ss.add(when: 'CONFIG_LIVE_BLOCK_MIGRATION', if_true: files('block.c')) softmmu_ss.add(when: zstd, if_true: files('multifd-zstd.c')) specific_ss.add(when: 'CONFIG_SOFTMMU', - if_true: files('dirtyrate.c', 'ram.c', 'target.c')) + if_true: files('dirtyrate.c', 'ram.c', 'target.c', 'confidential-ram.c'))
migration/trace-events +3 -0
@@ -344,3 +344,6 @@ migration_block_save_pending(uint64_t pending) "Enter save live pending %" PRIu # page_cache.c migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRId64 migration_pagecache_insert(void) "Error allocating page" + +# confidential-ram.c +encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64
Signature · 1 line
2.20.1
RFC PATCH v2 2/12 migration: Add helpers to load confidential RAM
In reply to Dov MurikQEMU cannot write directly to the memory of memory-encrypted guests; this breaks normal RAM-load in the migration target. Instead, QEMU asks a migration helper running on an auxiliary vcpu in the guest to restore encrypted pages as they were received from the source to a specific GPA.
The migration helper running inside the guest can safely decrypt the pages arrived from the source and load them into their proper location in the guest's memory.
Loading pages uses the same shared (unencrypted) pages which both QEMU and the guest can read from and write to.
--- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 37 ++++++++++++++++++++++++++++++++++++ migration/trace-events | 1 + 3 files changed, 40 insertions(+)
migration/confidential-ram.h +2 -0
@@ -14,4 +14,6 @@ void cgs_mh_cleanup(void);int cgs_mh_save_encrypted_page (QEMUFile *f,ram_addr_t src_gpa,uint32_t size,uint64_t *bytes_sent); +int cgs_mh_load_encrypted_page (QEMUFile *f,ram_addr_t dest_gpa); +#endif
migration/confidential-ram.c +37 -0
@@ -182,3 +182,40 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size,return ret; } + +int cgs_mh_load_encrypted_page (QEMUFile *f,ram_addr_t dest_gpa) +{ +int ret =1 ; +uint32_t page_hdr_len, enc_page_len; + +init_cgs_mig_helper_if_needed (); + +assert ((dest_gpa &TARGET_PAGE_MASK ) == dest_gpa); + +/* Read page header */ + page_hdr_len =qemu_get_be32 (f); +if (page_hdr_len >1024 ) { +error_report ("confidential-ram: page header is too large (%d bytes) " +"when loading gpa 0x%" PRIx64, page_hdr_len, dest_gpa); +return -EINVAL ; + } + cmhs.io_page_hdr->len = page_hdr_len; +qemu_get_buffer (f, cmhs.io_page_hdr->data, page_hdr_len); + +/* Read encrypted page */ + enc_page_len =qemu_get_be32 (f); +if (enc_page_len !=TARGET_PAGE_SIZE ) { +error_report ("confidential-ram: encrypted page is too large (%d bytes) " +"when loading gpa 0x%" PRIx64, enc_page_len, dest_gpa); +return -EINVAL ; + } +qemu_get_buffer (f, cmhs.io_page, enc_page_len); + +trace_encrypted_ram_load_page (page_hdr_len, enc_page_len, dest_gpa); + ret =send_command_to_cgs_mig_helper (CGS_MIG_HELPER_CMD_DECRYPT , dest_gpa); +if (ret) { +error_report ("confidential-ram: failed loading page at gpa " +"0x%" PRIx64": ret=%d" , dest_gpa, ret); + } +return ret; +}
migration/trace-events +1 -0
@@ -346,4 +346,5 @@ migration_pagecache_init(int64_t max_num_items) "Setting cache buckets to %" PRI migration_pagecache_insert(void) "Error allocating page" # confidential-ram.c +encrypted_ram_load_page(uint32_t hdr_len, uint32_t trans_len, uint64_t gpa) "hdr_len: %u, trans_len: %u, gpa: 0x%" PRIx64 encrypted_ram_save_page(uint32_t size, uint64_t gpa) "size: %u, gpa: 0x%" PRIx64
Signature · 1 line
2.20.1
RFC PATCH v2 3/12 migration: Introduce gpa_inside_migration_helper_shared_area
In reply to Dov MurikThe gpa_inside_migration_helper_shared_area will be used to skip migrating RAM pages that are used by the migration helper at the target.
--- migration/confidential-ram.h | 2 ++ migration/confidential-ram.c | 6 ++++++ 2 files changed, 8 insertions(+)
migration/confidential-ram.h +2 -0
@@ -8,6 +8,8 @@#include "exec/cpu-common.h" #include "qemu-file.h" +bool gpa_inside_migration_helper_shared_area (ram_addr_t gpa); +void cgs_mh_init (void );void cgs_mh_cleanup (void );
migration/confidential-ram.c +6 -0
@@ -68,6 +68,12 @@ static CGSMigHelperState cmhs = {0};#define MH_SHARED_CMD_PARAMS_ADDR 0x820000#define MH_SHARED_IO_PAGE_HDR_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x800)#define MH_SHARED_IO_PAGE_ADDR (MH_SHARED_CMD_PARAMS_ADDR + 0x1000) +#define MH_SHARED_LAST_BYTE (MH_SHARED_CMD_PARAMS_ADDR + 0x1fff) + +bool gpa_inside_migration_helper_shared_area (ram_addr_t gpa) +{ +return gpa >=MH_SHARED_CMD_PARAMS_ADDR && gpa <=MH_SHARED_LAST_BYTE ; +}void cgs_mh_init (void ) {
Signature · 1 line
2.20.1
RFC PATCH v2 4/12 migration: Save confidential guest RAM using migration helper
In reply to Dov MurikWhen saving RAM pages of a confidential guest, check whether a page is encrypted. If it is, ask the in-guest migration helper to encrypt the page for transmission.
This patch forces the use of in-guest migration handler instead of the PSP-based SEV migration; this is just a temporary example. TODO introduce migration parameter for choosing this migration mode.
--- include/sysemu/sev.h | 1 + migration/ram.c | 109 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 101 insertions(+), 9 deletions(-)
include/sysemu/sev.h +1 -0
@@ -19,6 +19,7 @@#define RAM_SAVE_ENCRYPTED_PAGE 0x1#define RAM_SAVE_SHARED_REGIONS_LIST 0x2 +#define RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE 0x4bool sev_enabled (void );int sev_kvm_init (ConfidentialGuestSupport *cgs,Error **errp);
migration/ram.c +100 -9
153 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH v2 5/12 migration: Load confidential guest RAM using migration helper
In reply to Dov MurikWhen loading encrypted RAM pages of a confidential guest, ask the in-guest migration helper to decrypt the incoming page and place it correctly in the guest memory at the appropriate address. This way the page's plaintext content remains inaccessible to the host.
--- migration/ram.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
migration/ram.c +16 -0
@@ -1250,6 +1250,7 @@ static int load_encrypted_data(QEMUFile *f, uint8_t *ptr) cgs_class->memory_encryption_ops;int flag; +hwaddr gpa; flag =qemu_get_be32 (f);
@@ -1257,6 +1258,12 @@ static int load_encrypted_data(QEMUFile *f, uint8_t *ptr)return ops->load_incoming_page (f, ptr); }else if (flag ==RAM_SAVE_SHARED_REGIONS_LIST ) {return ops->load_incoming_shared_regions_list (f); + }else if (flag ==RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE ) { +if (!kvm_physical_memory_addr_from_host (kvm_state, ptr, &gpa)) { +error_report ("%s: failed to get gpa for host ptr %p" , __func__, ptr); +return -EINVAL ; + } +return cgs_mh_load_encrypted_page (f, gpa); }else {error_report ("unknown encrypted flag %x" , flag);return 1 ;
@@ -3728,6 +3735,10 @@ void colo_release_ram_cache(void)*/ static int ram_load_setup (QEMUFile *f,void *opaque) { +if (confidential_guest ()) { +cgs_mh_init (); + } +if (compress_threads_load_setup (f)) {return -1 ; }
@@ -3754,6 +3765,10 @@ static int ram_load_cleanup(void *opaque) rb->receivedmap =NULL ; } +if (confidential_guest ()) { +cgs_mh_cleanup (); + } +return 0 ; }
@@ -4024,6 +4039,7 @@ void colo_flush_ram_cache(void)static int ram_load_precopy (QEMUFile *f) {int flags =0 , ret =0 , invalid_flags =0 , len =0 , i =0 ; +/* ADVISE is earlier, it shows the source has the postcopy capability on */ bool postcopy_advised =postcopy_is_advised ();if (!migrate_use_compression ()) {
Signature · 1 line
2.20.1
RFC PATCH v2 6/12 migration: Skip ROM, non-RAM, and vga.vram memory region during RAM migration
In reply to Dov MurikMigrating these memory region hangs the in-guest migration handler.
--- migration/ram.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
migration/ram.c +7 -1
@@ -2086,7 +2086,9 @@ static bool encrypted_test_list(RAMState *rs, RAMBlock *block,unsigned long gfn;/* ROM devices contains the unencrypted data */ -if (memory_region_is_rom (block->mr)) { +if (memory_region_is_rom (block->mr) || +memory_region_is_romd (block->mr) || + !memory_region_is_ram (block->mr)) {return false; }
@@ -2098,6 +2100,10 @@ static bool encrypted_test_list(RAMState *rs, RAMBlock *block,return false; } +if (!strcmp (memory_region_name (block->mr),"vga.vram" )) { +return false; + } +/* * Translate page in ram_addr_t address space to GPA address * space using memory region.
Signature · 1 line
2.20.1
RFC PATCH v2 7/12 i386/kvm: Exclude mirror vcpu in kvm_synchronize_all_tsc
In reply to Dov MurikIf we don't exclude it there's a hang when stopping the VM during migration.
--- target/i386/kvm/kvm.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
target/i386/kvm/kvm.c +3 -1
@@ -241,7 +241,9 @@ void kvm_synchronize_all_tsc(void)if (kvm_enabled ()) {CPU_FOREACH (cpu) { -run_on_cpu (cpu, do_kvm_synchronize_tsc,RUN_ON_CPU_NULL ); +if (!cpu->mirror_vcpu) { +run_on_cpu (cpu, do_kvm_synchronize_tsc,RUN_ON_CPU_NULL ); + } } } }
Signature · 1 line
2.20.1
RFC PATCH v2 8/12 migration: Allow resetting the mirror vcpu to the MH entry point
In reply to Dov MurikAdd a function to reset the mirror vcpu so it'll start directly at the entry point of the migration handler.
Note: In the patch below the GDT and EIP values are hard-coded to fit the OVMF migration handler entry point implementation we currently have. These values can be exposed in the OVMF GUID table and can be discovered from there instead of being hard-coded here.
--- migration/confidential-ram.h | 2 + migration/confidential-ram.c | 112 +++++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+)
migration/confidential-ram.h +2 -0
@@ -18,4 +18,6 @@ int cgs_mh_save_encrypted_page(QEMUFile *f, ram_addr_t src_gpa, uint32_t size,int cgs_mh_load_encrypted_page (QEMUFile *f,ram_addr_t dest_gpa); +void cgs_mh_reset_mirror_vcpu (CPUState *s); +#endif
migration/confidential-ram.c +112 -0
121 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH v2 9/12 migration: Add QMP command start-migration-handler
In reply to Dov MurikThe start-migration-handler QMP command starts the mirror vcpu directly at the migration handler entry point.
This is a temporary workaround to start-up (resume) the mirror vcpu which runs the in-guest migration handler (both on the source and the target).
A proper solution would be to start it automatically when the 'migrate' and 'migrate-incoming' QMP commands are executed.
--- qapi/migration.json | 12 ++++++++++++ migration/migration.c | 12 ++++++++++++ 2 files changed, 24 insertions(+)
qapi/migration.json +12 -0
@@ -1504,6 +1504,18 @@ ## { 'command': 'migrate-incoming', 'data': {'uri': 'str' } } +## +# @start-migration-handler: +# +# Start the mirror vcpu which runs the in-guest migration handler. +# +# Returns: nothing on success +# +# Since: 6.2 +# +## +{ 'command': 'start-migration-handler' } + ## # @xen-save-devices-state: #
migration/migration.c +12 -0
@@ -60,6 +60,7 @@#include "qemu/yank.h" #include "sysemu/cpus.h" #include "yank_functions.h" +#include "confidential-ram.h" #define MAX_THROTTLE (128 << 20)/* Migration transfer speed throttling */
@@ -2161,6 +2162,17 @@ void qmp_migrate_incoming(const char *uri, Error **errp) once = false; } +void qmp_start_migration_handler (Error **errp) +{ +CPUState *cpu; +CPU_FOREACH (cpu) { +if (cpu->mirror_vcpu) { +cgs_mh_reset_mirror_vcpu (cpu); +cpu_resume (cpu); + } + } +} +void qmp_migrate_recover (const char *uri,Error **errp) {MigrationIncomingState *mis =migration_incoming_get_current ();
Signature · 1 line
2.20.1
RFC PATCH v2 10/12 migration: Add start-migrate-incoming QMP command
In reply to Dov MurikThis command forces a running VM into a migrate-incoming state. When using guest-assisted migration (for confidential guests), the target must be started so that its memory has the necessary code for the migration helper. After it is ready we can start receiving the incoming migration connection.
--- qapi/migration.json | 26 ++++++++++++++++++++++++++ migration/migration.c | 17 +++++++++++++++++ 2 files changed, 43 insertions(+)
qapi/migration.json +26 -0
@@ -1516,6 +1516,32 @@ ## { 'command': 'start-migration-handler' } +## +# @start-migrate-incoming: +# +# Force start an incoming migration even in a running VM. This is used by the +# target VM in guest-assisted migration of a confidential guest. +# +# @uri: The Uniform Resource Identifier identifying the source or +# address to listen on +# +# Returns: nothing on success +# +# Since: 6.0 +# +# Notes: +# +# The uri format is the same as the -incoming command-line option. +# +# Example: +# +# -> { "execute": "start-migrate-incoming", +# "arguments": { "uri": "tcp::4446" } } +# <- { "return": {} } +# +## +{ 'command': 'start-migrate-incoming', 'data': {'uri': 'str' } } + ## # @xen-save-devices-state: #
migration/migration.c +17 -0
@@ -2173,6 +2173,23 @@ void qmp_start_migration_handler(Error **errp) } } +void qmp_start_migrate_incoming (const char *uri,Error **errp) +{ +Error *local_err =NULL ; + +if (!yank_register_instance (MIGRATION_YANK_INSTANCE , errp)) { +return ; + } + +vm_stop (RUN_STATE_PAUSED ); +qemu_start_incoming_migration (uri, &local_err); + +if (local_err) { +yank_unregister_instance (MIGRATION_YANK_INSTANCE ); +error_propagate (errp, local_err); + } +} +void qmp_migrate_recover (const char *uri,Error **errp) {MigrationIncomingState *mis =migration_incoming_get_current ();
Signature · 1 line
2.20.1
RFC PATCH v2 11/12 hw/isa/lpc_ich9: Allow updating an already-running VM
In reply to Dov MurikThe post_load function crashed when we were loading the device state in to an already-running guest. This was because an existing memory region as not deleted in ich9_lpc_rcba_update.
--- hw/isa/lpc_ich9.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
hw/isa/lpc_ich9.c +2 -1
@@ -527,9 +527,10 @@ ich9_lpc_pmcon_update(ICH9LPCState *lpc)static int ich9_lpc_post_load (void *opaque,int version_id) {ICH9LPCState *lpc = opaque; +uint32_t rcba_old =pci_get_long (lpc->d.config +ICH9_LPC_RCBA );ich9_lpc_pmbase_sci_update (lpc); -ich9_lpc_rcba_update (lpc,0 /* disabled ICH9_LPC_RCBA_EN */ ); +ich9_lpc_rcba_update (lpc, rcba_old);ich9_lpc_pmcon_update (lpc);return 0 ; }
Signature · 1 line
2.20.1
RFC PATCH v2 12/12 docs: Add confidential guest live migration documentation
In reply to Dov MurikThe new page is linked from the main index, otherwise sphinx complains that "document isn't included in any toctree"; I assume there would be a better place for it in the documentation tree.
--- docs/confidential-guest-live-migration.rst | 145 +++++++++++++++++++++ docs/confidential-guest-support.txt | 5 + docs/index.rst | 1 + 3 files changed, 151 insertions(+) create mode 100644 docs/confidential-guest-live-migration.rst
docs/confidential-guest-live-migration.rst new file +145 -0
@@ -0,0 +1,145 @@ +================================= +Confidential Guest Live Migration +================================= + +When migrating regular QEMU guests, QEMU reads the guest's RAM and sends it +over to the migration target host, where QEMU there writes it into the target +guest's RAM and starts the VM. This mechanism doesn't work when the guest +memory is encrypted or QEMU is prevented from reading it in another way. + +In order to support live migration in such scenarios, QEMU relies on an +in-guest migration helper which can securely extract RAM content from the +guest in order to send it to the target. The migration helper is implemented as +part of the VM's firmware in OVMF. + + +Migration flow +============== + +Source VM +--------- + +The source VM is started with an extra mirror vcpu which is not part of the +main VM but shared the same memory mapping. This vcpu is started at a special +entry point which runs a dedicated migration helper; the migration helper +simply waits for commands from QEMU. When migration starts using the +``migrate`` command, QEMU starts saving the state of the different devices. +When it reaches saving RAM pages, it'll check for each page whether it is +encrypted or not; for encrypted pages, it'll send a command to the migration +helper to extract the given page. The migration helper receives this command, +reads the page content, encrypts it with a transport key, and returns the +newly-encrypted page to QEMU. QEMU saves those pages to the outgoing migration +stream using the ``RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE`` subtype of the +``RAM_SAVE_FLAG_ENCRYPTED_DATA`` page flag. + +When QEMU reaches the last stage of RAM migration, it stops the source VM to +avoid dirtying the last pages of RAM. However, the mirror vcpu must be kept +running so the migration helper can still extract pages from the guest memory. + +Target VM +--------- + +Usually QEMU migration target VMs are started with the ``-incoming`` +command-line option which starts the VM paused. However, in order to migrate +confidential guests we must have the migration helper running inside the guest; +in such a case, we start the target with a special ``-fw_cfg`` value that tells +OVMF to load the migration handler code into memory and then enter a CPU dead +loop. After this short "boot" completes, QEMU can switch to the "migration +incoming" mode; we do that with the new ``start-migrate-incoming`` QMP command +that makes the target VM listen for incoming migration connections. + +QEMU will load the state of VM devices as it arrives from the incoming +migration stream. When it encounters a RAM page with the +``RAM_SAVE_FLAG_ENCRYPTED_DATA`` flag and the +``RAM_SAVE_GUEST_MH_ENCRYPTED_PAGE`` subtype, it will send its +transport-encrypted content and guest physical address to the migration helper. +The migration helper running inside the guest will decrypt the page using the +transport key and place the content in memory (again, that memory page is not +accessible to host due to the confidential guest properties; for example, in SEV +it is hardware-encrypted with a VM-specific key). + + +Usage +===== + +In order to start the source and target VMs with mirror vCPUs, the +``mirrorvcpus=`` option must be passed to ``-smp`` . For example:: + + # ${QEMU} -smp 5,mirrorvcpus=1 ... + +This command starts a VM with 5 vcpus of which 4 are main vcpus (available for +the guest OS) and 1 is mirror vcpu. + +Moreover, in both the source and target we need to instruct OVMF to start the +migration helper running in the auxiliary vcpu. This is achieved using the +following command-line option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationHelper,string=0 ... + +In the target VM we need to add another ``-fw_cfg`` entry to instruct OVMF to +start only the migration helepr, which will wait for incoming pages (the target +cannot be started with ``-incoming`` because that option completely pauses the +VM, not allowing the migration helper to run). Because the migration helper must +be running when the incoming RAM pages are received, starting the target VM with +the ``-incoming`` option doesn't work (with that option, the VM doesn't start +executing). Instead, start the target VM without ``-incoming`` but with the +following option:: + + # ${QEMU} -fw_cfg name=opt/ovmf/PcdSevIsMigrationTarget,string=1 ... + +After the VM boots into the migration helper, we instruct QEMU to listen for +incoming migration connections by sending the following QMP command:: + + { "execute": "start-migrate-incoming", + "arguments": { "uri": "tcp:0.0.0.0:6666" } } + +Now that the target is ready, we instruct the source VM to start migrating its +state using the regular ``migrate`` QMP command, supplying the target VMs +listening address:: + + { "execute": "migrate", + "arguments": { "uri": "tcp:192.168.111.222:6666" } } + + +Implementation details +====================== + +Migration helper <-> QEMU communication +--------------------------------------- + +The migration helper is running inside the guest (implemented as part of OVMF). +QEMU communicates with it using a mailbox protocol over two shared (unencrypted) +4K RAM pages. + +The first page contains a ``SevMigHelperCmdParams`` struct at offset 0x0 +(``cmd_params``) and a ``MigrationHelperHeader`` struct at offset 0x800 +(``io_hdr``). The second page (``io_page``) is dedicated for encrypted page +content. + +In order to save a confidential RAM page, QEMU will fill the ``cmd_params`` +struct to indicate the SEV_MIG_HELPER_CMD_ENCRYPT command and the requested gpa +(guest physical address), and then set the ``go`` field to 1. Meanwhile the +migration helper waits for the ``go`` field to become non-zero; after it notices +``go`` is 1 it'll read the gpa, read the content of the relevant page from the +guest's memory, encrypt it with the transport key, and store the +transport-encrypted page in the the ``io_page``. Additional envelope data like +encryption IV and other fields are stored in ``io_hdr``. After the migration is +done writing to ``io_page`` and ``io_hdr``, it sets the ``done`` field to 1. At +this point QEMU notices that the migration helper is done and can continue its +part, which is saving the header and page to the outgoing migration stream. + +Similar process is used when loading a confidential RAM from the incoming +migration stream. QEMU reads the header and the encrypted page from the stream, +and copies them into the shared areas ``io_hdr`` and ``io_page`` respectably. +It then fills the ``cmd_params`` struct to indicate the +SEV_MIG_HELPER_CMD_DECRYPT command and the gpa, and sets ``go`` to 1. The +migration helper will notice the command, will decrypt the page using the +transport key and will place the decrypted content in the requetsed gpa, and set +``done`` to 1 to allow QEMU to continue processing the next item in the incoming +migration stream. + +Shared pages address discovery +------------------------------ +In the current implementation the address of the two shared pages is hard-coded +in both OVMF and QEMU. We plan for OVMF to expose this address via its GUIDed +table and let QEMU discover it using ``pc_system_ovmf_table_find()``.
docs/confidential-guest-support.txt +5 -0
@@ -47,3 +47,8 @@ s390x Protected Virtualization (PV) docs/system/s390x/protvirt.rst Other mechanisms may be supported in future. + +Live migration support +---------------------- +Details regarding confidential guest live migration are in: + docs/confidential-guest-live-migration.rst
docs/index.rst +1 -0
@@ -17,3 +17,4 @@ Welcome to QEMU's documentation! interop/index specs/index devel/index + confidential-guest-live-migration
Signature · 1 line
2.20.1
Quoted 27 lines
-----Original Message-----
From: Qemu-devel [mailto:qemu-devel-bounces+shameerali.kolothum.thodi=huawei.com@nong nu.org] On Behalf Of Dov Murik
Sent: 23 August 2021 15:16
To: [email protected]
Cc: Tom Lendacky <[email protected]>; Ashish Kalra <[email protected]>; Brijesh Singh <[email protected]>; Michael S. Tsirkin <[email protected]>; Steve Rutherford <[email protected]>; James Bottomley <[email protected]>; Juan Quintela <[email protected]>; Dr. David Alan Gilbert <[email protected]>; Dov Murik <[email protected]>; Hubertus Franke <[email protected]>; Tobin Feldman-Fitzthum <[email protected]>; Paolo Bonzini <[email protected]>
Subject: [RFC PATCH v2 00/12] Confidential guest-assisted live migrationThis is an RFC series for fast migration of confidential guests using an in-guest migration helper that lives in OVMF. QEMU VM live migration needs to read source VM's RAM and write it in the target VM; this mechanism doesn't work when the guest memory is encrypted or QEMU is prevented from reading it in another way. In order to support live migration in such scenarios, we introduce an in-guest migration helper which can securely extract RAM content from the guest in order to send it to the target. The migration helper is implemented as part of the VM's firmware in OVMF.
Quoted 7 lines
We've implemented and tested this on AMD SEV, but expect most of the processes can be used with other technologies that prevent direct access of hypervisor to the guest's memory. Specifically, we don't use SEV's PSP migration commands (
SEV_SEND_START,SEV_RECEIVE_START, etc) at all; but note that the mirror VM relies onKVM_CAP_VM_COPY_ENC_CONTEXT_FROM
to shared the SEV ASID with the main VM.
Hi Dov,
Sorry if I missed out, but just to check if there are any updates to or revised one to this series? This guest-assisted method seems to be a good generic approach for live migration and just wondering whether it is worth taking a look for ARM CCA as well(I am not sure ARM RMM spec will have any specific proposal for live migration or not, but couldn't find anything public yet).
Please let me know if you plan to re-spin or there are any concerns with this approach. Appreciate if you can point me to any relevant discussion threads.
Thanks,
Shameer
Quoted 124 lines
Corresponding RFC patches for OVMF have been posted by Tobin Feldman-Fitzthum on edk2-devel [1]. Those include the crux of the migration helper: a mailbox protocol over a shared memory page which allows communication between QEMU and the migration helper. In the source VM this is used to read a page and encrypt it for transport; in the target it is used to decrypt the incoming page and storing the content in the correct address in the guest memory. All encryption and decryption operations occur inside the trusted context in the VM, and therefore the VM's memory plaintext content is never accessible to the hosts participating in the migration.
In order to allow OVMF to run the migration helper in parallel to the guest OS, we use a mirror VM [3], which shares the same memory mapping and SEV ASID as the main VM but has its own run loop. To start the mirror vcpu and the migration handler, we added a temporary start-migration-handler QMP command; this will be removed in a future version to run as part of the migrate QMP command.
In the target VM we need the migration handler running to receive incoming RAM pages; to achieve that, we boot the VM into OVMF with a special
fw_cfgvalue that causes OVMF to not boot the guest OS; we then allow QEMU to receive an incoming migration by issuing a new start-migrate-incoming QMP command.The confidential RAM migration requires checking whether a given guest RAM page is encrypted or not. This is achieved using SEV shared regions list tracking, which is implemented as part the SEV live migration patch series [2]. This feature tracks hypercalls from OVMF and guest Linux to report changes of page encryption status so that QEMU has an up-to-date view of which memory regions are shared and which are encrypted.
We left a few unfinished edges in this RFC but decided to publish it to start the commmunity discussion. TODOs:
1. QMP commands start-migration-handler and start-migrate-incoming are developer tools and should be performed automatically.
2. The entry point address of the in-guest migration handler and its GDT are currently hard-coded in QEMU (patch 8); instead they should be discovered usingpc_system_ovmf_table_find. Same applies for the mailbox address (patch 1).
3. For simplicity, this patch series forces the use of the guest-assisted migration instead of the SEV PSP-based migration. Ideally we might want the user to choose the desired mode using migrate-set-parameters or a similar mechanism.
4. There is currently no discovery protocol between QEMU and OVMF to verify that OVMF indeed supports in-guest migration handler.List of patches in this series:
1-3: introduce new confidtial RAM migration functions which communicate
with the migration helper.
4-6: use the new MH communication functions when migrating encrypted RAM
pages
7-9: allow starting migration handler on mirror vcpu with QMP command
start-migration-handler
10: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
11: fix devices issues when loading state into a live VM 12: add documentationThis patch series is based on top of:
1. Add SEV guest live migration support, from Ashish Kalra [2]
2. Support for mirror VM, from Ashish Kalra [3][1] https://edk2.groups.io/g/devel/message/79517
[2]
https://lore.kernel.org/qemu-devel/cover.1628076205.git.ashish.kalra@amd
.com/
[3]
https://lore.kernel.org/qemu-devel/cover.1629118207.git.ashish.kalra@amd
.com/Changes from RFC v1:
- Use the an SEV mirror VM for the migation handler (instead of auxilliary vcpus)RFC v1:
https://lore.kernel.org/qemu-devel/20210302204822.81901-1-dovmurik@li
nux.vnet.ibm.com/Dov Murik (12):
migration: Add helpers to save confidential RAM
migration: Add helpers to load confidential RAM
migration: Introducegpa_inside_migration_helper_shared_area
migration: Save confidential guest RAM using migration helper
migration: Load confidential guest RAM using migration helper
migration: Skip ROM, non-RAM, and vga.vram memory region during RAM
migration
i386/kvm: Exclude mirror vcpu inkvm_synchronize_all_tsc
migration: Allow resetting the mirror vcpu to the MH entry point
migration: Add QMP command start-migration-handler
migration: Add start-migrate-incoming QMP command hw/isa/lpc_ich9: Allow updating an already-running VM
docs: Add confidential guest live migration documentationdocs/confidential-guest-live-migration.rst | 145 +++++++++ docs/confidential-guest-support.txt | 5 + docs/index.rst | 1 + qapi/migration.json | 38 +++ include/sysemu/sev.h | 1 + migration/confidential-ram.h | 23 ++ hw/isa/lpc_ich9.c | 3 +- migration/confidential-ram.c | 339 +++++++++++++++++++++ migration/migration.c | 29 ++ migration/ram.c | 133 +++++++- target/i386/kvm/kvm.c | 4 +- migration/meson.build | 2 +- migration/trace-events | 4 + 13 files changed, 714 insertions(+), 13 deletions(-) create mode 100644 docs/confidential-guest-live-migration.rst create mode 100644 migration/confidential-ram.h create mode 100644 migration/confidential-ram.c--
2.20.1