Skip to contentSkip to conversation

Add import export API for confidential guest live migration

RFC PATCH Superseded Tony Lindgren 1 participant 4 messages Oct 6, 2025

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Add import export API for confidential guest live migration

Hi all,

This patch series is intended to get the discussion going on the APIs needed for confidential computing live migration for firmware related calls.

These APIs are designed to work for TDX, while also attempting to make them generic enough such that other coco VMs could use them. While the coco live migrations are still in flux, we wanted to see where the pain points are in making them work similarly from the userspace perspective. This way we can try to have at least some unification between all the solutions.

Currently live migration between the source and destination host machine is handled by the userspace with the help of KVM. The guest memory and VCPU states are accessible to the userspace to migrate. Userspace gets the list of dirty pages using the KVM_GET_DIRTY_LOG ioctl().

For coco VMs, things are different. The guest memory and VCPU states are not accessible to the userspace or KVM. Userspace may need to ask the source and destination host firmware to package and unpackage the guest memory and VCPU states for migration. The memory and VCPU states may be encrypted with a migration key between the source and destination firmware. Additionally the userspace may need to communicate with the host firmware to manage the live migration, and to pass migration related data between the source and destination firmware. For getting the list of dirty pages, KVM_GET_DIRTY_LOG is still used as earlier.

To allow handling coco VM live migration, introduce new ioctl() calls to communicate with the host firmware:

- KVM_MIGRATE_CMD
- KVM_IMPORT/EXPORT_MEMORY
- KVM_IMPORT/EXPORT_VCPU

No separate VM scope IMPORT/EXPORT calls are currently needed at least for TDX. It can be covered with the VM scope KVM_MIGRATE_CMD sub-commands.

Note that these calls are limited to cases where userspace needs to communicate with the hardware specific firmware. This may not apply to the service VM solutions like Coconut SVSM as it can access the memory and VCPU states directly.

To demonstrate how the userspace can make use of these calls, a simplified imaginary QEMU pseudo-code example for live migration is shown below:

1. QEMU source sets up migration

migration_thread()
migrate_cmd(MIGRATE_SETUP, data)
...
kvm_arch_vm_ioctl(KVM_MIGRATE_CMD, KVM_MIGRATE_SETUP, data)
arch_migrate_cmd(MIGRATE_SETUP, data)

2. QEMU destination sets up migration

migrate_cmd(MIGRATE_SETUP, data)
...
kvm_arch_vm_ioctl(KVM_MIGRATE_CMD, KVM_MIGRATE_SETUP, data)
arch_migrate_cmd(MIGRATE_SETUP, data)

QEMU return-path can be used as needed.

3. QEMU source saves memory

ram_save_target_page()
ram_export_memory(data)
...
kvm_arch_vm_ioctl(KVM_EXPORT_MEMORY, data)
arch_export_memory(data)

4. QEMU destination loads memory

ram_load_precopy()
ram_import_memory(data)
...
kvm_arch_vm_ioctl(KVM_IMPORT_MEMORY, data)
arch_import_memory(data)

5. QEMU source and destination configure additional VCPU VMState

static const VMStateInfo vcpu_state_info = {
.name = "private-vcpu"
.put = export_vcpu_state
.get = import_vcpu_state
}

6. QEMU source saves VCPU state

vmstate_save_state()
export_vcpu_state(data)
...
kvm_vcpu_arch_ioctl(KVM_EXPORT_VCPU, data)
arch_export_vcpu_state(data)

7. QEMU destination loads VCPU state
vmstate_load_state()
import_vcpu_state(data)
...
kvm_vcpu_arch_ioctl(KVM_IMPORT_VCPU, data)
arch_import_vcpu_state(data)

8. QEMU source migration finish

migration_iteration_finish()
migrate_cmd(KVM_MIGRATE_CMD, MIGRATE_FINISH, data)
...
kvm_arch_vm_ioctl(KVM_MIGRATE_CMD, MIGRATE_FINISH, data)
arch_migrate_cmd(MIGRATE_FINISH, data)

Regards,

Tony

Tony Lindgren (3):
Documentation: kvm: Add KVM_MIGRATE_CMD
Documentation: kvm: Add KVM_IMPORT/EXPORT_MEMORY
Documentation: kvm: Add KVM_IMPORT/EXPORT_VCPU

 Documentation/virt/kvm/api.rst | 203 +++++++++++++++++++++++++++++++++
 1 file changed, 203 insertions(+)

base-commit: 6b36119b94d0b2bb8cea9d512017efafd461d6ac

Signature · 1 line
2.43.0
#1 lore raw

RFC PATCH 1/3 Documentation: kvm: Add KVM_MIGRATE_CMD

Document KVM_MIGRATE_CMD. To support live migration of confidential computing guests, the hardware may need to pass migration command related data between the source and destination. For confidential computing, the migration command related data is not accessible to KVM.

---
 Documentation/virt/kvm/api.rst | 47 ++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
Documentation/virt/kvm/api.rst +47 -0
index 6ae24c5ca559..31db949d3e44 100644
@@ -6504,6 +6504,53 @@ the capability to be present.
 `flags` must currently be zero.
 
 
+4.144 KVM_MIGRATE_CMD
+---------------------
+
+:Capability: KVM_CAP_MIGRATION
+:Architectures: arm64, x86
+:Type: vm ioctl
+:Parameters: struct kvm_migrate_cmd (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to send migration commands to the hardware.
+
+For confidential computing, the migration commands may use encrypted data
+that needs to be passed between the source and destination hardware. The
+hardware may also require specific coordination steps during migration that
+must be triggered at precise points in the migration process.
+
+The parameter related data structures are::
+
+  #define KVM_MIGRATE_PREPARE		0
+  #define KVM_MIGRATE_SETUP		1
+  #define KVM_MIGRATE_TOKEN		2
+  #define KVM_MIGRATE_SOURCE_BLACKOUT	3
+  #define KVM_MIGRATE_ABORT		4
+  #define KVM_MIGRATE_FINISH		5
+
+  struct kvm_transfer_buffer {
+	__u64 address;
+	__u32 size;
+	__u32 reserved;
+  };
+
+  @address  - Userspace buffer address
+  @size     - Size of the userspace buffer
+  @reserved - Reserved for future use
+
+  struct kvm_migrate_cmd {
+	__u16 command;
+	__u16 flags;
+	__u32 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+  @command  - One of the defined KVM_MIGRATE commands
+  @flags    - Hardware specific flags
+  @reserved - Reserved for future use
+  @buf      - Userspace buffer for hardware specific data
+
 .. _kvm_run:
 
 5. The kvm_run structure
Signature · 1 line
2.43.0
#2 lore raw

RFC PATCH 2/3 Documentation: kvm: Add KVM_IMPORT/EXPORT_MEMORY

In reply to Tony Lindgren

Document KVM_IMPORT_MEMORY and KVM_EXPORT_MEMORY. To support live migration of confidential computing guests, the hardware needs to export the encrypted pages on the source and to import the encrypted pages on the destination.

---
 Documentation/virt/kvm/api.rst | 89 +++++++++++++++++++++++++++++++++-
 1 file changed, 88 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +88 -1
index 31db949d3e44..dec73fd2c5bf 100644
@@ -6503,7 +6503,6 @@ the capability to be present.
 
 `flags` must currently be zero.
 
-
 4.144 KVM_MIGRATE_CMD
 ---------------------
 
@@ -6551,6 +6550,94 @@ The parameter related data structures are::
   @reserved - Reserved for future use
   @buf      - Userspace buffer for hardware specific data
 
+.. _KVM_IMPORT_MEMORY:
+
+4.145 KVM_IMPORT_MEMORY
+-----------------------
+
+:Capability: KVM_CAP_MIGRATION
+:Architectures: arm64, x86
+:Type: vm ioctl
+:Parameters: struct kvm_memory_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the hardware to import an array of memory pages from
+a userspace buffer.
+
+The memory may not be directly accessible to KVM because of encryption. For
+confidential computing, the guest memory is encrypted and only accessible to the
+guest.
+
+The parameter related data structures are::
+
+  struct kvm_transfer_buffer {
+	__u64 address;
+	__u32 size;
+	__u32 reserved;
+  };
+
+  @address  - Userspace buffer address
+  @size     - Size of the userspace buffer
+  @reserved - Reserved for future use
+
+  struct kvm_memory_transfer {
+	__u64 gfns;
+	__u32 nr_gfns;
+	__u16 id;
+	__u16 flags;
+	__u64 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+  @gfns     - Userspace array of GFNs to import
+  @nr_gfns  - Number of GFNs
+  @id       - Optional hardware specific transfer ID
+  @flags    - Hardware specific flags
+  @reserved - Reserved for future use
+  @buf      - Userspace buffer to import memory from
+
+The hardware specific ID is used at least for TDX for the migration thread
+index.
+
+4.146 KVM_EXPORT_MEMORY
+-----------------------
+
+:Capability: KVM_CAP_MIGRATION
+:Architectures: arm64, x86
+:Type: vm ioctl
+:Parameters: struct kvm_memory_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the hardware to export an array of memory pages
+to a userspace buffer.
+
+The memory may not be directly accessible to KVM because of encryption. For
+confidential computing, the guest memory is encrypted and only accessible to the
+guest.
+
+The parameters are::
+
+  struct kvm_memory_transfer {
+	__u64 gfns;
+	__u32 nr_gfns;
+	__u16 id;
+	__u16 flags;
+	__u64 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+  @gfns     - Userspace array of GFNs to export
+  @nr_gfns  - Number of GFNs
+  @id       - Optional hardware specific transfer ID
+  @flags    - Hardware specific flags
+  @reserved - Reserved for future use
+  @buf      - Userspace buffer to export memory to
+
+The hardware specific ID is used at least for TDX for the migration thread
+index.
+
+See also :ref:`KVM_IMPORT_MEMORY <KVM_IMPORT_MEMORY>`.
+
 .. _kvm_run:
 
 5. The kvm_run structure
Signature · 1 line
2.43.0
#3 lore raw

RFC PATCH 3/3 Documentation: kvm: Add KVM_IMPORT/EXPORT_VCPU

In reply to Tony Lindgren

Document KVM_IMPORT_VCPU and KVM_EXPORT_VCPU. To support live migration of confidential computing guests, the hardware needs to export the encrypted VCPU state on the source and to import the encrypted VCPU state on the destination.

---
 Documentation/virt/kvm/api.rst | 69 ++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)
Documentation/virt/kvm/api.rst +69 -0
index dec73fd2c5bf..d9499e3b461a 100644
@@ -6638,6 +6638,75 @@ index.
 
 See also :ref:`KVM_IMPORT_MEMORY <KVM_IMPORT_MEMORY>`.
 
+.. _KVM_IMPORT_VCPU:
+
+4.147 KVM_IMPORT_VCPU
+---------------------
+
+:Capability: KVM_CAP_MIGRATION
+:Architectures: arm64, x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_vcpu_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the hardware to import a VCPU state from a userspace
+buffer.
+
+The VCPU state may not be directly accessible to KVM because of encryption. For
+confidential computing, the VCPU state is encrypted and only accessible to the
+guest.
+
+The parameter related data structures are::
+
+  struct kvm_transfer_buffer {
+	__u64 address;
+	__u32 size;
+	__u32 reserved;
+  };
+
+  @address  - Userspace buffer address
+  @size     - Size of the userspace buffer
+  @reserved - Reserved for future use
+
+  struct kvm_vcpu_transfer {
+        __u32 flags;
+        __u32 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+  @flags    - Hardware specific flags
+  @reserved - Reserved for future use
+  @buf      - Userspace buffer to import VCPU state from
+
+4.148 KVM_EXPORT_VCPU
+---------------------
+:Capability: KVM_CAP_MIGRATION
+:Architectures: arm64, x86
+:Type: vcpu ioctl
+:Parameters: struct kvm_vcpu_transfer (in/out)
+:Returns: 0 on success, < 0 on error
+
+Allows userspace to request the hardware to export a VCPU state to a userspace
+buffer.
+
+The VCPU state may not be directly accessible to KVM because of encryption. For
+confidential computing, the VCPU state is encrypted and only accessible to the
+guest.
+
+The parameters are::
+
+  struct kvm_vcpu_transfer {
+        __u32 flags;
+        __u32 reserved;
+	struct kvm_transfer_buffer buf;
+  };
+
+  @flags    - Hardware specific flags
+  @reserved - Reserved for future use
+  @buf      - Userspace buffer to export VCPU state to
+
+See also :ref:`KVM_IMPORT_VCPU <KVM_IMPORT_VCPU>`.
+
 .. _kvm_run:
 
 5. The kvm_run structure
Signature · 1 line
2.43.0