Introduce support for confidential guest reset (x86)
v3 cover Introduce support for confidential guest reset (x86)
This change introduces support for confidential guests
(SEV-ES, SEV-SNP and TDX) to reset/reboot just like other non-confidential guests. Currently, a reboot intiated from the confidential guest results in termination of the QEMU hypervisor as the CPUs are not resettable. As the initial state of the guest including private memory is locked and encrypted, the contents of that memory will not be accessible post reset. Hence a new KVM file descriptor must be opened to create a new confidential VM context closing the old one. All KVM VM specific ioctls must be called again. New VCPU file descriptors must be created against the new KVM fd and most VCPU ioctls must be called again as well.
This change perfoms closing of the old KVM fd and creating a new one. After the new KVM fd is opened, all generic and architecture specific ioctl calls are issued again. Notifiers are added to notify subsystems that:
- The KVM file fd is about to be changed to state sync-ing from KVM to QEMU should be done if required.
- The KVM file fd has changed, so ioctl calls to the new KVM fd has to be performed again.
- That new VCPU fds are created so that VCPU ioctl calls must be called again where required.
Specific subsystems use these notifiers to re-issue ioctl calls where required.
Changes are made to SEV and TDX modules to reinitialize the confidential guest state and seal it again. Along the way, some bug fixes are made so that some initialization functions can be called again. Some refactoring of existing code is done so that both init and reset paths can use them.
Tested on TDX, SEV-ES and SEV-SNP. Tested on non-coco hardware. Tested with Xen emulation enabled.
CI pipeline passes:
https://gitlab.com/anisinha/qemu/-/pipelines/2278323424
Please review and test.
Changelog:
v3:
- Combined pre and post vmfd change notifier into one.
- rename kvm_arch_vmfd_change_ops() -> kvm_arch_on_vmfd_change()
- reuse kvm_arch_init() code in kvm_arch_on_vmfd_change()
- moved around migration blockers and notifers to more appropriate place.
- fixed Xen emulation.
- fixed SEV-ES reset.
- fixed/reorganized reset code in system/runstate.c
- can_rebuild_guest_state is now a boolean not a callback.
- misc fixes.
- added a functional test for Xen emulation with vmfd change.
- rebased.
v2:
- Bugfixes.
- Added a new machine option so that we can exercize most of the non-coco changes related to reboot on non-coco platforms.
- added a new functional test. Currently its skipped on CI pipeline as KVM is not enabled (no /dev/kvm on the container)for QEMU CI tests. It can be run manually and it passes on those systems where KVM is enabled.
- Addressed comments from v1 with regards to refactoring of code, code simplication by removal of redundant stuff, moved around code
so that notifiers, migration blockers are added only on one place.
- Added some tracepoints for future debugging on newly added functions.
- Rebased.
Ani Sinha (33):
i386/kvm: avoid installing duplicate msr entries in msr_handlers accel/kvm: add confidential class member to indicate guest rebuild
capability
hw/accel: add a per-accelerator callback to change VM accelerator
handle
system/physmem: add helper to reattach existing memory after KVM VM fd
change
accel/kvm: add changes required to support KVM VM file descriptor
change
accel/kvm: mark guest state as unprotected after vm file descriptor
change
accel/kvm: add a notifier to indicate KVM VM file descriptor has
changed
accel/kvm: notify when KVM VM file fd is about to be changed i386/kvm: unregister smram listeners prior to vm file descriptor
change
kvm/i386: implement architecture support for kvm file descriptor
change
hw/i386: refactor x86_bios_rom_init for reuse in confidential guest
reset
hw/i386: export a new function x86_bios_rom_reload kvm/i386: reload firmware for confidential guest reset accel/kvm: rebind current VCPUs to the new KVM VM file descriptor upon
reset
i386/tdx: refactor TDX firmware memory initialization code into a new
function
i386/tdx: finalize TDX guest state upon reset
i386/tdx: add a pre-vmfd change notifier to reset tdx state i386/sev: add migration blockers only once
i386/sev: add notifiers only once
i386/sev: free existing launch update data and kernel hashes data on
init
i386/sev: add support for confidential guest reset hw/vfio: generate new file fd for pseudo device and rebind existing
descriptors
kvm/i8254: refactor pit initialization into a helper kvm/i8254: add support for confidential guest reset hw/hyperv/vmbus: add support for confidential guest reset kvm/xen-emu: re-initialize capabilities during confidential guest
reset
kvm/xen_evtchn: add support for confidential guest reset ppc/openpic: create a new openpic device and reattach mem region on
coco reset
kvm/vcpu: add notifiers to inform vcpu file descriptor change kvm/i386/apic: set local apic after vcpu file descriptors changed kvm/clock: add support for confidential guest reset hw/machine: introduce machine specific option 'x-change-vmfd-on-reset' tests/functional/x86_64: add functional test to exercise vm fd change
on reset
MAINTAINERS | 6 + accel/kvm/kvm-all.c | 348 ++++++++++++++++-- accel/kvm/trace-events | 2 + accel/stubs/kvm-stub.c | 18 + hw/core/machine.c | 22 ++ hw/hyperv/vmbus.c | 36 ++ hw/i386/kvm/apic.c | 13 + hw/i386/kvm/clock.c | 59 +++ hw/i386/kvm/i8254.c | 88 +++-- hw/i386/kvm/xen_evtchn.c | 113 +++++- hw/i386/x86-common.c | 72 +++- hw/intc/openpic_kvm.c | 112 ++++-- hw/vfio/helpers.c | 86 ++++- include/accel/accel-ops.h | 1 + include/hw/core/boards.h | 6 + include/hw/i386/apic_internal.h | 1 + include/hw/i386/x86.h | 2 +- include/system/confidential-guest-support.h | 20 + include/system/kvm.h | 42 +++ include/system/physmem.h | 1 + system/physmem.c | 28 ++ system/runstate.c | 30 +- target/arm/kvm.c | 10 + target/i386/kvm/kvm.c | 137 +++++-- target/i386/kvm/tdx.c | 141 +++++-- target/i386/kvm/tdx.h | 1 + target/i386/kvm/trace-events | 4 + target/i386/kvm/xen-emu.c | 50 ++- target/i386/sev.c | 127 +++++-- target/i386/trace-events | 1 + target/loongarch/kvm/kvm.c | 10 + target/mips/kvm.c | 10 + target/ppc/kvm.c | 10 + target/riscv/kvm/kvm-cpu.c | 10 + target/s390x/kvm/kvm.c | 10 + tests/functional/x86_64/meson.build | 1 + .../x86_64/test_vmfd_change_reboot.py | 96 +++++ 37 files changed, 1508 insertions(+), 216 deletions(-) create mode 100755 tests/functional/x86_64/test_vmfd_change_reboot.py
Signature · 1 line
2.42.0
PATCH v3 1/33 i386/kvm: avoid installing duplicate msr entries in msr_handlers
kvm_filter_msr() does not check if an msr entry is already present in the msr_handlers table and installs a new handler unconditionally. If the function is called again with the same MSR, it will result in duplicate entries in the table and multiple such calls will fill up the table needlessly. Fix that.
--- target/i386/kvm/kvm.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
target/i386/kvm/kvm.c +16 -10
@@ -6043,27 +6043,33 @@ static int kvm_install_msr_filters(KVMState *s)static int kvm_filter_msr (KVMState *s,uint32_t msr,QEMURDMSRHandler *rdmsr,QEMUWRMSRHandler *wrmsr) { -int i, ret; +int i, ret =0 ;for (i =0 ; i <ARRAY_SIZE (msr_handlers); i++) { -if (!msr_handlers[i].msr) { +if (msr_handlers[i].msr == msr) { +break ; + }else if (!msr_handlers[i].msr) { msr_handlers[i] = (KVMMSRHandlers ) { .msr = msr, .rdmsr = rdmsr, .wrmsr = wrmsr, }; +break ; + } + } - ret =kvm_install_msr_filters (s); -if (ret) { - msr_handlers[i] = (KVMMSRHandlers ) { }; -return ret; - } +if (i ==ARRAY_SIZE (msr_handlers)) { + ret = -EINVAL ; + goto end; + } -return 0 ; - } + ret =kvm_install_msr_filters (s); +if (ret) { + msr_handlers[i] = (KVMMSRHandlers ) { }; } -return -EINVAL ; + end: +return ret; }static int kvm_handle_rdmsr (X86CPU *cpu,struct kvm_run *run)
Signature · 1 line
2.42.0
PATCH v3 2/33 accel/kvm: add confidential class member to indicate guest rebuild capability
In reply to Ani SinhaAs a part of the confidential guest reset process, the existing encrypted guest state must be made mutable since it would be discarded after reset. A new encrypted and locked guest state must be established after the reset. To this end, a new boolean member per confidential guest support class (eg, tdx or sev-snp) is added that will indicate whether its possible to rebuild guest state:
bool can_rebuild_guest_state;
This is true if rebuilding guest state is possible, false otherwise. A KVM based confidential guest reset is only possible when the existing state is locked but its possible to rebuild guest state. Otherwise, the guest is not resettable.
--- include/system/confidential-guest-support.h | 20 ++++++++++++++++++++ system/runstate.c | 6 +++--- target/i386/kvm/tdx.c | 1 + target/i386/sev.c | 1 + 4 files changed, 25 insertions(+), 3 deletions(-)
include/system/confidential-guest-support.h +20 -0
@@ -152,6 +152,11 @@ typedef struct ConfidentialGuestSupportClass {*/ int (*get_mem_map_entry)(int index,ConfidentialGuestMemoryMapEntry *entry,Error **errp); + +/* +* is it possible to rebuild the guest state? +*/ +bool can_rebuild_guest_state; } ConfidentialGuestSupportClass; static inline int confidential_guest_kvm_init(ConfidentialGuestSupport *cgs,
@@ -167,6 +172,21 @@ static inline int confidential_guest_kvm_init(ConfidentialGuestSupport *cgs,return 0 ; } +static inline bool +confidential_guest_can_rebuild_state (ConfidentialGuestSupport *cgs) +{ +ConfidentialGuestSupportClass *klass; + +if (!cgs) { +/* non-confidential guests */ +return true; + } + + klass =CONFIDENTIAL_GUEST_SUPPORT_GET_CLASS (cgs); +return klass->can_rebuild_guest_state; + +} +static inline int confidential_guest_kvm_reset (ConfidentialGuestSupport *cgs,Error **errp) {
system/runstate.c +3 -3
@@ -57,6 +57,7 @@#include "system/reset.h" #include "system/runstate.h" #include "system/runstate-action.h" +#include "system/confidential-guest-support.h" #include "system/system.h" #include "system/tpm.h" #include "trace.h"
@@ -543,8 +544,6 @@ void qemu_system_reset(ShutdownCause reason)*/ if (cpus_are_resettable ()) {cpu_synchronize_all_post_reset (); - }else { -assert (runstate_check (RUN_STATE_PRELAUNCH )); }vm_set_suspended (false);
@@ -693,7 +692,8 @@ void qemu_system_reset_request(ShutdownCause reason)if (reboot_action ==REBOOT_ACTION_SHUTDOWN && reason !=SHUTDOWN_CAUSE_SUBSYSTEM_RESET ) { shutdown_requested = reason; - }else if (!cpus_are_resettable ()) { + }else if (!cpus_are_resettable () && + !confidential_guest_can_rebuild_state (current_machine->cgs)) {error_report ("cpus are not resettable, terminating" ); shutdown_requested = reason; }else {
target/i386/kvm/tdx.c +1 -0
@@ -1543,6 +1543,7 @@ static void tdx_guest_class_init(ObjectClass *oc, const void *data)X86ConfidentialGuestClass *x86_klass =X86_CONFIDENTIAL_GUEST_CLASS (oc); klass->kvm_init = tdx_kvm_init; + klass->can_rebuild_guest_state = true; x86_klass->kvm_type = tdx_kvm_type; x86_klass->cpu_instance_init = tdx_cpu_instance_init; x86_klass->adjust_cpuid_features = tdx_adjust_cpuid_features;
target/i386/sev.c +1 -0
@@ -2760,6 +2760,7 @@ sev_common_instance_init(Object *obj) cgs->set_guest_state = cgs_set_guest_state; cgs->get_mem_map_entry = cgs_get_mem_map_entry; cgs->set_guest_policy = cgs_set_guest_policy; + cgs->can_rebuild_guest_state = true;QTAILQ_INIT (&sev_common->launch_vmsa); }
Signature · 1 line
2.42.0
PATCH v3 3/33 hw/accel: add a per-accelerator callback to change VM accelerator handle
In reply to Ani SinhaWhen a confidential virtual machine is reset, a new guest context in the accelerator must be generated post reset. Therefore, the old accelerator guest file handle must closed and a new one created. To this end, a per-accelerator callback, "reset_vmfd" is introduced that would get called when a confidential guest is reset. Subsequent patches will introduce specific implementation of this callback for KVM accelerator.
--- include/accel/accel-ops.h | 1 + system/runstate.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
include/accel/accel-ops.h +1 -0
@@ -23,6 +23,7 @@ struct AccelClass {AccelOpsClass *ops;int (*init_machine)(AccelState *as,MachineState *ms); +int (*reset_vmfd)(MachineState *ms);bool (*cpu_common_realize)(CPUState *cpu,Error **errp);void (*cpu_common_unrealize)(CPUState *cpu);/* get_stats: Append statistics to @buf */
system/runstate.c +23 -0
@@ -42,6 +42,7 @@#include "qapi/qapi-commands-run-state.h" #include "qapi/qapi-events-run-state.h" #include "qemu/accel.h" +#include "accel/accel-ops.h" #include "qemu/error-report.h" #include "qemu/job.h" #include "qemu/log.h"
@@ -509,6 +510,8 @@ void qemu_system_reset(ShutdownCause reason) {MachineClass *mc;ResetType type; +AccelClass *ac =ACCEL_GET_CLASS (current_accel ()); +int ret; mc = current_machine ?MACHINE_GET_CLASS (current_machine) :NULL ;
@@ -521,6 +524,26 @@ void qemu_system_reset(ShutdownCause reason)default : type =RESET_TYPE_COLD ; } + +/* +* different accelerators implement how to close the old file handle of +* the accelerator descriptor and create a new one here. Resetting +* file handle is necessary to create a new confidential VM context post +* VM reset. +*/ +if (!cpus_are_resettable () && reason ==SHUTDOWN_CAUSE_GUEST_RESET ) { +if (ac->reset_vmfd) { + ret = ac->reset_vmfd (current_machine); +if (ret <0 ) { +error_report ("unable to reset vmfd: %s(%d)" , +strerror (-ret), ret); +vm_stop (RUN_STATE_INTERNAL_ERROR ); + } + }else { +error_report ("accelerator does not support reset" ); + } + } +if (mc && mc->reset) { mc->reset (current_machine, type); }else {
Signature · 1 line
2.42.0
PATCH v3 4/33 system/physmem: add helper to reattach existing memory after KVM VM fd change
In reply to Ani SinhaAfter the guest KVM file descriptor has changed as a part of the process of confidential guest reset mechanism, existing memory needs to be reattached to the new file descriptor. This change adds a helper function ram_block_rebind() for this purpose. The next patch will make use of this function.
--- include/system/physmem.h | 1 + system/physmem.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+)
include/system/physmem.h +1 -0
@@ -51,5 +51,6 @@ physical_memory_snapshot_and_clear_dirty(MemoryRegion *mr, hwaddr offset,bool physical_memory_snapshot_get_dirty (DirtyBitmapSnapshot *snap,ram_addr_t start,ram_addr_t length); +int ram_block_rebind (Error **errp);#endif
system/physmem.c +28 -0
@@ -2857,6 +2857,34 @@ found:return block; } +/* +* Creates new guest memfd for the ramblocks and closes the +* existing memfd. +*/ +int ram_block_rebind (Error **errp) +{ +RAMBlock *block; + +qemu_mutex_lock_ramlist (); + +RAMBLOCK_FOREACH (block) { +if (block->flags &RAM_GUEST_MEMFD ) { +if (block->guest_memfd >=0 ) { +close (block->guest_memfd); + } + block->guest_memfd =kvm_create_guest_memfd (block->max_length, +0 , errp); +if (block->guest_memfd <0 ) { +qemu_mutex_unlock_ramlist (); +return -1 ; + } + + } + } +qemu_mutex_unlock_ramlist (); +return 0 ; +} +/* * Finds the named RAMBlock *
Signature · 1 line
2.42.0
PATCH v3 5/33 accel/kvm: add changes required to support KVM VM file descriptor change
In reply to Ani SinhaThis change adds common kvm specific support to handle KVM VM file descriptor change. KVM VM file descriptor can change as a part of confidential guest reset mechanism. A new function api kvm_arch_on_vmfd_change() per architecture platform is added in order to implement architecture specific changes required to support it. A subsequent patch will add x86 specific implementation for kvm_arch_on_vmfd_change() as currently only x86 supports confidential guest reset.
--- accel/kvm/kvm-all.c | 80 ++++++++++++++++++++++++++++++++++++-- accel/kvm/trace-events | 1 + include/system/kvm.h | 2 + target/arm/kvm.c | 10 +++++ target/i386/kvm/kvm.c | 10 +++++ target/loongarch/kvm/kvm.c | 10 +++++ target/mips/kvm.c | 10 +++++ target/ppc/kvm.c | 10 +++++ target/riscv/kvm/kvm-cpu.c | 10 +++++ target/s390x/kvm/kvm.c | 10 +++++ 10 files changed, 150 insertions(+), 3 deletions(-)
accel/kvm/kvm-all.c +77 -3
@@ -2415,11 +2415,9 @@ void kvm_irqchip_set_qemuirq_gsi(KVMState *s, qemu_irq irq, int gsi)g_hash_table_insert (s->gsimap, irq,GINT_TO_POINTER (gsi)); } -static void kvm_irqchip_create (KVMState *s) +static void do_kvm_irqchip_create (KVMState *s) {int ret; - -assert (s->kernel_irqchip_split !=ON_OFF_AUTO_AUTO );if (kvm_check_extension (s,KVM_CAP_IRQCHIP )) { ; }else if (kvm_check_extension (s,KVM_CAP_S390_IRQCHIP )) {
@@ -2452,7 +2450,13 @@ static void kvm_irqchip_create(KVMState *s)fprintf (stderr,"Create kernel irqchip failed: %s\n" ,strerror (-ret));exit (1 ); } +} +static void kvm_irqchip_create (KVMState *s) +{ +assert (s->kernel_irqchip_split !=ON_OFF_AUTO_AUTO ); + +do_kvm_irqchip_create (s); kvm_kernel_irqchip = true;/* If we have an in-kernel IRQ chip then we must have asynchronous * interrupt delivery (though the reverse is not necessarily true)
@@ -2607,6 +2611,75 @@ static int kvm_setup_dirty_ring(KVMState *s)return 0 ; } +static int kvm_reset_vmfd (MachineState *ms) +{ +KVMState *s; +KVMMemoryListener *kml; +int ret =0 , type; +Error *err =NULL ; + +/* +* bail if the current architecture does not support VM file +* descriptor change. +*/ +if (!kvm_arch_supports_vmfd_change ()) { +error_report ("This target architecture does not support KVM VM " +"file descriptor change." ); +return -EOPNOTSUPP ; + } + + s =KVM_STATE (ms->accelerator); + kml = &s->memory_listener; + +memory_listener_unregister (&kml->listener); +memory_listener_unregister (&kvm_io_listener); + +if (s->vmfd >=0 ) { +close (s->vmfd); + } + + type =find_kvm_machine_type (ms); +if (type <0 ) { +return -EINVAL ; + } + + ret =do_kvm_create_vm (s, type); +if (ret <0 ) { +return ret; + } + + s->vmfd = ret; + +kvm_setup_dirty_ring (s); + +/* rebind memory to new vm fd */ + ret =ram_block_rebind (&err); +if (ret <0 ) { +return ret; + } +assert (!err); + + ret =kvm_arch_on_vmfd_change (ms, s); +if (ret <0 ) { +return ret; + } + +if (s->kernel_irqchip_allowed) { +do_kvm_irqchip_create (s); + } + +/* these can be only called after ram_block_rebind() */ +memory_listener_register (&kml->listener, &address_space_memory); +memory_listener_register (&kvm_io_listener, &address_space_io); + +/* +* kvm fd has changed. Commit the irq routes to KVM once more. +*/ +kvm_irqchip_commit_routes (s); +trace_kvm_reset_vmfd (); +return ret; +} +static int kvm_init (AccelState *as,MachineState *ms) {MachineClass *mc =MACHINE_GET_CLASS (ms);
@@ -4014,6 +4087,7 @@ static void kvm_accel_class_init(ObjectClass *oc, const void *data)AccelClass *ac =ACCEL_CLASS (oc); ac->name ="KVM" ; ac->init_machine = kvm_init; + ac->reset_vmfd = kvm_reset_vmfd; ac->has_memory = kvm_accel_has_memory; ac->allowed = &kvm_allowed; ac->gdbstub_supported_sstep_flags = kvm_gdbstub_sstep_flags;
accel/kvm/trace-events +1 -0
@@ -14,6 +14,7 @@ kvm_destroy_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu" kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu" kvm_unpark_vcpu(unsigned long arch_cpu_id, const char *msg) "id: %lu %s" kvm_irqchip_commit_routes(void) "" +kvm_reset_vmfd(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d" kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d" kvm_irqchip_release_virq(int virq) "virq %d"
include/system/kvm.h +2 -0
@@ -358,6 +358,8 @@ int kvm_arch_init(MachineState *ms, KVMState *s);int kvm_arch_pre_create_vcpu (CPUState *cpu,Error **errp);int kvm_arch_init_vcpu (CPUState *cpu);int kvm_arch_destroy_vcpu (CPUState *cpu); +bool kvm_arch_supports_vmfd_change (void ); +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s);#ifdef TARGET_KVM_HAVE_RESET_PARKED_VCPU void kvm_arch_reset_parked_vcpu (unsigned long vcpu_id,int kvm_fd);
target/arm/kvm.c +10 -0
@@ -1569,6 +1569,16 @@ void kvm_arch_init_irq_routing(KVMState *s) { } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +int kvm_arch_irqchip_create (KVMState *s) {if (kvm_kernel_irqchip_split ()) {
target/i386/kvm/kvm.c +10 -0
@@ -3253,6 +3253,16 @@ static int kvm_vm_enable_energy_msrs(KVMState *s)return 0 ; } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {int ret;
target/loongarch/kvm/kvm.c +10 -0
@@ -1364,6 +1364,16 @@ int kvm_arch_irqchip_create(KVMState *s)return kvm_check_extension (s,KVM_CAP_DEVICE_CTRL ); } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +void kvm_arch_pre_run (CPUState *cs,struct kvm_run *run) { }
target/mips/kvm.c +10 -0
@@ -44,6 +44,16 @@ unsigned long kvm_arch_vcpu_id(CPUState *cs)return cs->cpu_index; } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +int kvm_arch_init (MachineState *ms,KVMState *s) {/* MIPS has 128 signals */
target/ppc/kvm.c +10 -0
@@ -180,6 +180,16 @@ int kvm_arch_irqchip_create(KVMState *s)return 0 ; } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +static int kvm_arch_sync_sregs (PowerPCCPU *cpu) {CPUPPCState *cenv = &cpu->env;
target/riscv/kvm/kvm-cpu.c +10 -0
@@ -1545,6 +1545,16 @@ int kvm_arch_irqchip_create(KVMState *s)return kvm_check_extension (s,KVM_CAP_DEVICE_CTRL ); } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +int kvm_arch_process_async_events (CPUState *cs) {return 0 ;
target/s390x/kvm/kvm.c +10 -0
@@ -393,6 +393,16 @@ int kvm_arch_irqchip_create(KVMState *s)return 0 ; } +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +unsigned long kvm_arch_vcpu_id (CPUState *cpu) {return cpu->cpu_index;
Signature · 1 line
2.42.0
Re: [PATCH v3 05/33] accel/kvm: add changes required to support KVM VM file descriptor change
Quoted 6 lines
target/arm/kvm.c +0
-0@@ -1569,6 +1569,16 @@ void kvm_arch_init_irq_routing(KVMState *s) { }
target/arm/kvm.c void kvm_arch_init_irq_routing(KVMState *s) · line 1581
Quoted 10 lines
target/arm/kvm.c +10
-0+int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) +{ +abort (); +} + +bool kvm_arch_supports_vmfd_change (void ) +{ +return false; +} +
Put that into stubs/kvm.c ? Should avoid the duplication for all archs which do not support vmfd_change.
take care,
Gerd
PATCH v3 6/33 accel/kvm: mark guest state as unprotected after vm file descriptor change
In reply to Ani SinhaWhen the KVM VM file descriptor has changed and a new one created, the guest state is no longer in protected state. Mark it as such.
The guest state becomes protected again when TDX and SEV-ES and SEV-SNP mark it as such.
--- accel/kvm/kvm-all.c | 3 +++ 1 file changed, 3 insertions(+)
accel/kvm/kvm-all.c +3 -0
@@ -2650,6 +2650,9 @@ static int kvm_reset_vmfd(MachineState *ms) s->vmfd = ret; +/* guest state is now unprotected again */ + kvm_state->guest_state_protected = false; +kvm_setup_dirty_ring (s);/* rebind memory to new vm fd */
Signature · 1 line
2.42.0
PATCH v3 7/33 accel/kvm: add a notifier to indicate KVM VM file descriptor has changed
In reply to Ani SinhaA notifier callback can be used by various subsystems to perform actions when KVM file descriptor for a virtual machine changes as a part of confidential guest reset process. This change adds this notifier mechanism. Subsequent patches will add specific implementations for various notifier callbacks corresponding to various subsystems that need to take action when KVM VM file descriptor changed.
--- accel/kvm/kvm-all.c | 30 ++++++++++++++++++++++++++++++ accel/stubs/kvm-stub.c | 8 ++++++++ include/system/kvm.h | 21 +++++++++++++++++++++ 3 files changed, 59 insertions(+)
accel/kvm/kvm-all.c +30 -0
@@ -90,6 +90,7 @@ struct KVMParkedVcpu { };KVMState *kvm_state; +VmfdChangeNotifier vmfd_notifier;bool kvm_kernel_irqchip;bool kvm_split_irqchip;bool kvm_async_interrupts_allowed;
@@ -123,6 +124,9 @@ static const KVMCapabilityInfo kvm_required_capabilites[] = {static NotifierList kvm_irqchip_change_notifiers =NOTIFIER_LIST_INITIALIZER (kvm_irqchip_change_notifiers); +static NotifierWithReturnList register_vmfd_changed_notifiers = +NOTIFIER_WITH_RETURN_LIST_INITIALIZER (register_vmfd_changed_notifiers); +struct KVMResampleFd {int gsi;EventNotifier *resample_event;
@@ -2173,6 +2177,22 @@ void kvm_irqchip_change_notify(void)notifier_list_notify (&kvm_irqchip_change_notifiers,NULL ); } +void kvm_vmfd_add_change_notifier (NotifierWithReturn *n) +{ +notifier_with_return_list_add (®ister_vmfd_changed_notifiers, n); +} + +void kvm_vmfd_remove_change_notifier (NotifierWithReturn *n) +{ +notifier_with_return_remove (n); +} + +static int kvm_vmfd_change_notify (Error **errp) +{ +return notifier_with_return_list_notify (®ister_vmfd_changed_notifiers, + &vmfd_notifier, errp); +} +int kvm_irqchip_get_virq (KVMState *s) {int next_virq;
@@ -2671,6 +2691,16 @@ static int kvm_reset_vmfd(MachineState *ms)do_kvm_irqchip_create (s); } +/* +* notify everyone that vmfd has changed. +*/ + vmfd_notifier.vmfd = s->vmfd; + ret =kvm_vmfd_change_notify (&err); +if (ret <0 ) { +return ret; + } +assert (!err); +/* these can be only called after ram_block_rebind() */ memory_listener_register (&kml->listener, &address_space_memory);memory_listener_register (&kvm_io_listener, &address_space_io);
accel/stubs/kvm-stub.c +8 -0
@@ -79,6 +79,14 @@ void kvm_irqchip_change_notify(void) { } +void kvm_vmfd_add_change_notifier (NotifierWithReturn *n) +{ +} + +void kvm_vmfd_remove_change_notifier (NotifierWithReturn *n) +{ +} +int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n,EventNotifier *rn,int virq) {
include/system/kvm.h +21 -0
@@ -181,6 +181,7 @@ DECLARE_INSTANCE_CHECKER(KVMState, KVM_STATE,extern KVMState *kvm_state;typedef struct Notifier Notifier ; +typedef struct NotifierWithReturn NotifierWithReturn ; typedef struct KVMRouteChange { KVMState *s;
@@ -566,4 +567,24 @@ int kvm_set_memory_attributes_shared(hwaddr start, uint64_t size);int kvm_convert_memory (hwaddr start,hwaddr size,bool to_private); +/* argument to vmfd change notifier */ +typedef struct VmfdChangeNotifier { +int vmfd; +}VmfdChangeNotifier ; + +/** +* kvm_vmfd_add_change_notifier - register a notifier to get notified when +* a KVM vm file descriptor changes as a part of the confidential guest "reset" +* process. Various subsystems should use this mechanism to take actions such +* as creating new fds against this new vm file descriptor. +* @n: notifier with return value. +*/ +void kvm_vmfd_add_change_notifier (NotifierWithReturn *n); +/** +* kvm_vmfd_remove_change_notifier - de-register a notifer previously +* registered with kvm_vmfd_add_change_notifier call. +* @n: notifier that was previously registered. +*/ +void kvm_vmfd_remove_change_notifier (NotifierWithReturn *n); +#endif
Signature · 1 line
2.42.0
PATCH v3 8/33 accel/kvm: notify when KVM VM file fd is about to be changed
In reply to Ani SinhaVarious subsystems might need to take some steps before the KVM file descriptor for a virtual machine is changed. So a new boolean attribute is added to the vmfd_notifier structure which is passed to the notifier callbacks. vmfd_notifer.pre is true for pre-notification of vmfd change and false for post notification. Notifier callback implementations can simply check the boolean value for (vmfd_notifer*)->pre and can take actions for pre or post vmfd change based on the value.
Subsequent patches will add callback implementations for specific components that need this pre-notification.
--- accel/kvm/kvm-all.c | 9 +++++++++ include/system/kvm.h | 6 ++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
accel/kvm/kvm-all.c +9 -0
@@ -2654,6 +2654,13 @@ static int kvm_reset_vmfd(MachineState *ms)memory_listener_unregister (&kml->listener);memory_listener_unregister (&kvm_io_listener); + vmfd_notifier.pre = true; + ret =kvm_vmfd_change_notify (&err); +if (ret <0 ) { +return ret; + } +assert (!err); +if (s->vmfd >=0 ) {close (s->vmfd); }
@@ -2695,6 +2702,8 @@ static int kvm_reset_vmfd(MachineState *ms)* notify everyone that vmfd has changed. */ vmfd_notifier.vmfd = s->vmfd; + vmfd_notifier.pre = false; + ret =kvm_vmfd_change_notify (&err);if (ret <0 ) {return ret;
include/system/kvm.h +4 -2
@@ -570,12 +570,14 @@ int kvm_convert_memory(hwaddr start, hwaddr size, bool to_private);/* argument to vmfd change notifier */ typedef struct VmfdChangeNotifier {int vmfd; +bool pre; }VmfdChangeNotifier ;/** * kvm_vmfd_add_change_notifier - register a notifier to get notified when -* a KVM vm file descriptor changes as a part of the confidential guest "reset" -* process. Various subsystems should use this mechanism to take actions such +* a KVM vm file descriptor changes or about to be changed as a part of the +* confidential guest "reset" process. +* Various subsystems should use this mechanism to take actions such * as creating new fds against this new vm file descriptor. * @n: notifier with return value. */
Signature · 1 line
2.42.0
PATCH v3 9/33 i386/kvm: unregister smram listeners prior to vm file descriptor change
In reply to Ani SinhaWe will re-register smram listeners after the VM file descriptors has changed. We need to unregister them first to make sure addresses and reference counters work properly.
--- target/i386/kvm/kvm.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
target/i386/kvm/kvm.c +18 -0
@@ -113,6 +113,11 @@ typedef struct {static void kvm_init_msrs (X86CPU *cpu);static int kvm_filter_msr (KVMState *s,uint32_t msr,QEMURDMSRHandler *rdmsr,QEMUWRMSRHandler *wrmsr); +static int unregister_smram_listener (NotifierWithReturn *notifier, +void *data,Error ** errp); +NotifierWithReturn kvm_vmfd_change_notifier = { + .notify = unregister_smram_listener, +};const KVMCapabilityInfo kvm_arch_required_capabilities[] = {KVM_CAP_INFO (SET_TSS_ADDR ),
@@ -2749,6 +2754,17 @@ static void register_smram_listener(Notifier *n, void *unused) } } +static int unregister_smram_listener (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +if (!((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +memory_listener_unregister (&smram_listener.listener); +return 0 ; +} +/* It should only be called in cpu's hotplug callback */ void kvm_smm_cpu_address_space_init (X86CPU *cpu) {
@@ -3401,6 +3417,8 @@ int kvm_arch_init(MachineState *ms, KVMState *s) } } +kvm_vmfd_add_change_notifier (&kvm_vmfd_change_notifier); +return 0 ; }
Signature · 1 line
2.42.0
PATCH v3 10/33 kvm/i386: implement architecture support for kvm file descriptor change
In reply to Ani SinhaWhen the kvm file descriptor changes as a part of confidential guest reset, some architecture specific setups including SEV/SEV-SNP/TDX specific setups needs to be redone. These changes are implemented as a part of the kvm_arch_on_vmfd_change() callback which was introduced previously.
--- target/i386/kvm/kvm.c | 82 ++++++++++++++++++++++++++---------- target/i386/kvm/trace-events | 1 + 2 files changed, 60 insertions(+), 23 deletions(-)
target/i386/kvm/kvm.c +59 -23
@@ -3269,14 +3269,52 @@ static int kvm_vm_enable_energy_msrs(KVMState *s)return 0 ; } +static int xen_init_wrapper (MachineState *ms,KVMState *s); +int kvm_arch_on_vmfd_change (MachineState *ms,KVMState *s) { -abort (); +int ret; + + ret =kvm_arch_init (ms, s); +if (ret <0 ) { +return ret; + } + +if (object_dynamic_cast (OBJECT (ms),TYPE_X86_MACHINE )) { +X86MachineState *x86ms =X86_MACHINE (ms); + +if (x86_machine_is_smm_enabled (x86ms)) { +memory_listener_register (&smram_listener.listener, + &smram_address_space); + } +kvm_set_max_apic_id (x86ms->apic_id_limit); + } + +trace_kvm_arch_on_vmfd_change (); +return 0 ; +} + +static int xen_init_wrapper (MachineState *ms,KVMState *s) +{ +int ret =0 ; +#ifdef CONFIG_XEN_EMU +if (!object_dynamic_cast (OBJECT (ms),TYPE_PC_MACHINE )) { +error_report ("kvm: Xen support only available in PC machine" ); +return -ENOTSUP ; + } +/* hyperv_enabled() doesn't work yet. */ +uint32_t msr =XEN_HYPERCALL_MSR ; + ret =kvm_xen_init (s, msr); +#else +error_report ("kvm: Xen support not enabled in qemu" ); +return -ENOTSUP ; +#endif +return ret; }bool kvm_arch_supports_vmfd_change (void ) { -return false; +return true; }int kvm_arch_init (MachineState *ms,KVMState *s)
@@ -3284,6 +3322,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s)int ret;struct utsname utsname;Error *local_err =NULL ; +static bool first = true;/* * Initialize confidential guest (SEV/TDX) context, if required
@@ -3312,21 +3351,10 @@ int kvm_arch_init(MachineState *ms, KVMState *s) }if (s->xen_version) { -#ifdef CONFIG_XEN_EMU -if (!object_dynamic_cast (OBJECT (ms),TYPE_PC_MACHINE )) { -error_report ("kvm: Xen support only available in PC machine" ); -return -ENOTSUP ; - } -/* hyperv_enabled() doesn't work yet. */ -uint32_t msr =XEN_HYPERCALL_MSR ; - ret =kvm_xen_init (s, msr); + ret =xen_init_wrapper (ms, s);if (ret <0 ) {return ret; } -#else -error_report ("kvm: Xen support not enabled in qemu" ); -return -ENOTSUP ; -#endif } ret =kvm_get_supported_msrs (s);
@@ -3353,16 +3381,17 @@ int kvm_arch_init(MachineState *ms, KVMState *s)return ret; } -/* Tell fw_cfg to notify the BIOS to reserve the range. */ -e820_add_entry (KVM_IDENTITY_BASE ,0x4000 ,E820_RESERVED ); - +if (first) { +/* Tell fw_cfg to notify the BIOS to reserve the range. */ +e820_add_entry (KVM_IDENTITY_BASE ,0x4000 ,E820_RESERVED ); + } ret =kvm_vm_set_nr_mmu_pages (s);if (ret <0 ) {return ret; }if (object_dynamic_cast (OBJECT (ms),TYPE_X86_MACHINE ) && -x86_machine_is_smm_enabled (X86_MACHINE (ms))) { +x86_machine_is_smm_enabled (X86_MACHINE (ms)) && first) { smram_machine_done.notify = register_smram_listener;qemu_add_machine_init_done_notifier (&smram_machine_done); }
@@ -3409,15 +3438,22 @@ int kvm_arch_init(MachineState *ms, KVMState *s)return ret; } - ret =kvm_msr_energy_thread_init (s, ms); -if (ret <0 ) { -error_report ("kvm : error RAPL feature requirement not met" ); -return ret; +if (first) { + ret =kvm_msr_energy_thread_init (s, ms); +if (ret <0 ) { +error_report ("kvm : " +"error RAPL feature requirement not met" ); +return ret; + } } } } -kvm_vmfd_add_change_notifier (&kvm_vmfd_change_notifier); +if (first) { +kvm_vmfd_add_change_notifier (&kvm_vmfd_change_notifier); + } + + first = false;return 0 ; }
target/i386/kvm/trace-events +1 -0
@@ -6,6 +6,7 @@ kvm_x86_add_msi_route(int virq) "Adding route entry for virq %d" kvm_x86_remove_msi_route(int virq) "Removing route entry for virq %d" kvm_x86_update_msi_routes(int num) "Updated %d MSI routes" kvm_hc_map_gpa_range(uint64_t gpa, uint64_t size, uint64_t attributes, uint64_t flags) "gpa 0x%" PRIx64 " size 0x%" PRIx64 " attributes 0x%" PRIx64 " flags 0x%" PRIx64 +kvm_arch_on_vmfd_change(void) "" # xen-emu.c kvm_xen_hypercall(int cpu, uint8_t cpl, uint64_t input, uint64_t a0, uint64_t a1, uint64_t a2, uint64_t ret) "xen_hypercall: cpu %d cpl %d input %" PRIu64 " a0 0x%" PRIx64 " a1 0x%" PRIx64 " a2 0x%" PRIx64" ret 0x%" PRIx64
Signature · 1 line
2.42.0
PATCH v3 11/33 hw/i386: refactor x86_bios_rom_init for reuse in confidential guest reset
In reply to Ani SinhaFor confidential guests, bios image must be reinitialized upon reset. This is because bios memory is encrypted and hence once the old confidential kvm context is destroyed, it cannot be decrypted. It needs to be reinitilized. In order to do that, this change refactors x86_bios_rom_init() code so that parts of it can be called during confidential guest reset.
--- hw/i386/x86-common.c | 51 ++++++++++++++++++++++++++++++++----------- include/hw/i386/x86.h | 1 - 2 files changed, 38 insertions(+), 14 deletions(-)
hw/i386/x86-common.c +38 -13
@@ -1024,17 +1024,11 @@ void x86_isa_bios_init(MemoryRegion *isa_bios, MemoryRegion *isa_memory,memory_region_set_readonly (isa_bios, read_only); } -void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware, -MemoryRegion *rom_memory,bool isapc_ram_fw) +static int get_bios_size (X86MachineState *x86ms, +const char *bios_name,char *filename) { -const char *bios_name; -char *filename;int bios_size; -ssize_t ret; -/* BIOS load */ - bios_name =MACHINE (x86ms)->firmware ?: default_firmware; - filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name);if (filename) { bios_size =get_image_size (filename,NULL ); }else {
@@ -1044,6 +1038,21 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware, (bios_size % 65536) != 0) { goto bios_error; } + +return bios_size; + + bios_error: +fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); +exit (1 ); +} + +static void load_bios_from_file (X86MachineState *x86ms,const char *bios_name, +char *filename,int bios_size, +bool isapc_ram_fw) +{ +ssize_t ret; + +/* BIOS load */ if (machine_require_guest_memfd (MACHINE (x86ms))) {memory_region_init_ram_guest_memfd (&x86ms->bios,NULL ,"pc.bios" , bios_size, &error_fatal);
@@ -1072,7 +1081,26 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware, goto bios_error; } } -g_free (filename); + +return ; + + bios_error: +fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); +exit (1 ); +} + +void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware, +MemoryRegion *rom_memory,bool isapc_ram_fw) +{ +int bios_size; +const char *bios_name; +char *filename; + + bios_name =MACHINE (x86ms)->firmware ?: default_firmware; + filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name); + + bios_size =get_bios_size (x86ms, bios_name, filename); +load_bios_from_file (x86ms, bios_name, filename, bios_size, isapc_ram_fw);if (!machine_require_guest_memfd (MACHINE (x86ms))) {/* map the last 128KB of the BIOS in ISA space */
@@ -1084,9 +1112,6 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,memory_region_add_subregion (rom_memory, (uint32_t )(-bios_size), &x86ms->bios); +g_free (filename);return ; - -bios_error: -fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); -exit (1 ); }
include/hw/i386/x86.h +0 -1
@@ -122,7 +122,6 @@ void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp);void x86_cpu_unplug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp); -void x86_isa_bios_init (MemoryRegion *isa_bios,MemoryRegion *isa_memory,MemoryRegion *bios,bool read_only); void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,
Signature · 1 line
2.42.0
Re: [PATCH v3 11/33] hw/i386: refactor x86_bios_rom_init for reuse in confidential guest reset
Am 27. Januar 2026 05:15:39 UTC schrieb Ani Sinha <[email protected]>:
Quoted 87 lines
For confidential guests, bios image must be reinitialized upon reset. This is because bios memory is encrypted and hence once the old confidential kvm context is destroyed, it cannot be decrypted. It needs to be reinitilized. In order to do that, this change refactors
x86_bios_rom_init()code so that parts of it can be called during confidential guest reset.Signed-off-by Ani Sinha <[email protected]>---
hw/i386/x86-common.c| 51 ++++++++++++++++++++++++++++++++-----------include/hw/i386/x86.h| 1 -2 files changed, 38 insertions(+), 14 deletions(-)hw/i386/x86-common.c +37
-9@@ -1024,17 +1024,11 @@ void x86_isa_bios_init(MemoryRegion *isa_bios, MemoryRegion *isa_memory,memory_region_set_readonly (isa_bios, read_only); } -void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware, -MemoryRegion *rom_memory,bool isapc_ram_fw) +static int get_bios_size (X86MachineState *x86ms, +const char *bios_name,char *filename) { -const char *bios_name; -char *filename;int bios_size; -ssize_t ret; -/* BIOS load */ - bios_name =MACHINE (x86ms)->firmware ?: default_firmware; - filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name);if (filename) { bios_size =get_image_size (filename,NULL ); }else {@@ -1044,6 +1038,21 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware, (bios_size % 65536) != 0) { goto bios_error; } + +return bios_size; + + bios_error: +fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); +exit (1 ); +} + +static void load_bios_from_file (X86MachineState *x86ms,const char *bios_name, +char *filename,int bios_size, +bool isapc_ram_fw) +{ +ssize_t ret; + +/* BIOS load */ if (machine_require_guest_memfd (MACHINE (x86ms))) {memory_region_init_ram_guest_memfd (&x86ms->bios,NULL ,"pc.bios" , bios_size, &error_fatal);@@ -1072,7 +1081,26 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware, goto bios_error; } } -g_free (filename); + +return ; + + bios_error: +fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); +exit (1 ); +} + +void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware, +MemoryRegion *rom_memory,bool isapc_ram_fw) +{ +int bios_size; +const char *bios_name; +char *filename; + + bios_name =MACHINE (x86ms)->firmware ?: default_firmware; + filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name); + + bios_size =get_bios_size (x86ms, bios_name, filename); +load_bios_from_file (x86ms, bios_name, filename, bios_size, isapc_ram_fw);if (!machine_require_guest_memfd (MACHINE (x86ms))) {/* map the last 128KB of the BIOS in ISA space */
hw/i386/x86-common.c void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware, · line 1115
Quoted 4 lines
hw/i386/x86-common.c +1
-0@@ -1084,9 +1112,6 @@ void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,memory_region_add_subregion (rom_memory, (uint32_t )(-bios_size), &x86ms->bios); +g_free (filename);
Doesn't the error path leak filename? Using g_autofree would prevent that. Even if the error path exits immediately today it may represent a latent bug for tomorrow (e.g. when porting to error API).
Quoted 11 lines
hw/i386/x86-common.c +0
-4return ; - -bios_error: -fprintf (stderr,"qemu: could not load PC BIOS '%s'\n" , bios_name); -exit (1 ); }include/hw/i386/x86.h +0
-0
include/hw/i386/x86.h void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, · line 125
Quoted 4 lines
include/hw/i386/x86.h +0
-1@@ -122,7 +122,6 @@ void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp);void x86_cpu_unplug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp); -
This file contains whitespace changes only and breaks logic grouping of functions. I'd drop this change.
Best regards,
Bernhard
Quoted 3 lines
include/hw/i386/x86.h +0
-0void x86_isa_bios_init (MemoryRegion *isa_bios,MemoryRegion *isa_memory,MemoryRegion *bios,bool read_only); void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,
PATCH v3 12/33 hw/i386: export a new function x86_bios_rom_reload
In reply to Ani SinhaConfidential guest smust reload their bios rom upon reset. This is because bios memory is encrypted and upon reset, the contents of the old bios memory is lost and cannot be re-used. To this end, export a new x86 function x86_bios_rom_reload() to reload the bios again. This function will be used in the subsequent patches.
--- hw/i386/x86-common.c | 21 +++++++++++++++++++++ include/hw/i386/x86.h | 1 + 2 files changed, 22 insertions(+)
hw/i386/x86-common.c +21 -0
@@ -1089,6 +1089,27 @@ static void load_bios_from_file(X86MachineState *x86ms, const char *bios_name,exit (1 ); } +void x86_bios_rom_reload (X86MachineState *x86ms) +{ +int bios_size; +const char *bios_name; +char *filename; + +if (!x86ms->bios.addr) { +/* if -bios is not used */ +return ; + } + + bios_name =MACHINE (x86ms)->firmware ?:"bios.bin" ; + filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name); + + bios_size =get_bios_size (x86ms, bios_name, filename); + +void *ptr =memory_region_get_ram_ptr (&x86ms->bios); +load_image_size (filename, ptr, bios_size); +x86_firmware_configure (0x100000000ULL - bios_size, ptr, bios_size); +} +void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware,MemoryRegion *rom_memory,bool isapc_ram_fw) {
include/hw/i386/x86.h +1 -0
@@ -122,6 +122,7 @@ void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp);void x86_cpu_unplug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp); +void x86_bios_rom_reload (X86MachineState *x86ms);void x86_isa_bios_init (MemoryRegion *isa_bios,MemoryRegion *isa_memory,MemoryRegion *bios,bool read_only); void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,
Signature · 1 line
2.42.0
Re: [PATCH v3 12/33] hw/i386: export a new function x86_bios_rom_reload
Am 27. Januar 2026 05:15:40 UTC schrieb Ani Sinha <[email protected]>:
Quoted 17 lines
Confidential guest smust reload their bios rom upon reset. This is because bios memory is encrypted and upon reset, the contents of the old bios memory is lost and cannot be re-used. To this end, export a new x86 function
x86_bios_rom_reload()to reload the bios again. This function will be used in the subsequent patches.Signed-off-by Ani Sinha <[email protected]>---
hw/i386/x86-common.c| 21 +++++++++++++++++++++include/hw/i386/x86.h| 1 +2 files changed, 22 insertions(+)hw/i386/x86-common.c +0
-0
hw/i386/x86-common.c static void load_bios_from_file(X86MachineState *x86ms, const char *bios_name, · line 1098
Quoted 10 lines
hw/i386/x86-common.c +7
-0@@ -1089,6 +1089,27 @@ static void load_bios_from_file(X86MachineState *x86ms, const char *bios_name,exit (1 ); } +void x86_bios_rom_reload (X86MachineState *x86ms) +{ +int bios_size; +const char *bios_name; +char *filename; + +if (!x86ms->bios.addr) {
Could we use memory_region_size() here instead to rely only on public API? AFAIU .addr is a private field.
Quoted 22 lines
hw/i386/x86-common.c +14
-0+/* if -bios is not used */ +return ; + } + + bios_name =MACHINE (x86ms)->firmware ?:"bios.bin" ; + filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , bios_name); + + bios_size =get_bios_size (x86ms, bios_name, filename); + +void *ptr =memory_region_get_ram_ptr (&x86ms->bios); +load_image_size (filename, ptr, bios_size); +x86_firmware_configure (0x100000000ULL - bios_size, ptr, bios_size); +} +void x86_bios_rom_init (X86MachineState *x86ms,const char *default_firmware,MemoryRegion *rom_memory,bool isapc_ram_fw) {include/hw/i386/x86.h +0
-0
include/hw/i386/x86.h void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, · line 125
Quoted 4 lines
include/hw/i386/x86.h +1
-0@@ -122,6 +122,7 @@ void x86_cpu_unplug_request_cb(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp);void x86_cpu_unplug_cb (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp); +void x86_bios_rom_reload (X86MachineState *x86ms);
Yes, this looks much cleaner!
To preserve the logic grouping I'd keep the blank line here from my comment in the previous patch. Additionally, I'd move the declaration after x86_bios_rom_init() to also communicate the order in which the functions are intended to be used.
With the above considered and the declaration moved the code seems comprehensible to me, so:
Quoted 3 lines
include/hw/i386/x86.h +0
-0void x86_isa_bios_init (MemoryRegion *isa_bios,MemoryRegion *isa_memory,MemoryRegion *bios,bool read_only); void x86_bios_rom_init(X86MachineState *x86ms, const char *default_firmware,
PATCH v3 13/33 kvm/i386: reload firmware for confidential guest reset
In reply to Ani SinhaWhen IGVM is not being used by the confidential guest, the guest firmware has to be reloaded explictly again into memory. This is because, the memory into which the firmware was loaded before reset was encrypted and is thus lost upon reset. When IGVM is used, it is expected that the IGVM will contain the guest firmware and the execution of the IGVM directives will set up the guest firmware memory.
--- target/i386/kvm/kvm.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
target/i386/kvm/kvm.c +8 -1
@@ -3282,7 +3282,14 @@ int kvm_arch_on_vmfd_change(MachineState *ms, KVMState *s)if (object_dynamic_cast (OBJECT (ms),TYPE_X86_MACHINE )) {X86MachineState *x86ms =X86_MACHINE (ms); - +/* +* For confidential guests, reload bios ROM if IGVM is not specified. +* If an IGVM file is specified then the firmware must be provided +* in the IGVM file. +*/ +if (ms->cgs && !x86ms->igvm) { +x86_bios_rom_reload (x86ms); + }if (x86_machine_is_smm_enabled (x86ms)) {memory_listener_register (&smram_listener.listener, &smram_address_space);
Signature · 1 line
2.42.0
PATCH v3 14/33 accel/kvm: rebind current VCPUs to the new KVM VM file descriptor upon reset
In reply to Ani SinhaConfidential guests needs to generate a new KVM file descriptor upon virtual machine reset. Existing VCPUs needs to be reattached to this new KVM VM file descriptor. As a part of this, new VCPU file descriptors against this new KVM VM file descriptor needs to be created and re-initialized. Resources allocated against the old VCPU fds needs to be released. This change makes this happen.
--- accel/kvm/kvm-all.c | 200 +++++++++++++++++++++++++++++++++-------- accel/kvm/trace-events | 1 + 2 files changed, 166 insertions(+), 35 deletions(-)
accel/kvm/kvm-all.c +165 -35
@@ -127,6 +127,10 @@ static NotifierList kvm_irqchip_change_notifiers =static NotifierWithReturnList register_vmfd_changed_notifiers =NOTIFIER_WITH_RETURN_LIST_INITIALIZER (register_vmfd_changed_notifiers); +static int map_kvm_run (KVMState *s,CPUState *cpu,Error **errp); +static int map_kvm_dirty_gfns (KVMState *s,CPUState *cpu,Error **errp); +static int vcpu_unmap_regions (KVMState *s,CPUState *cpu); +struct KVMResampleFd {int gsi;EventNotifier *resample_event;
@@ -420,6 +424,83 @@ err:return ret; } +static int kvm_rebind_vcpus (Error **errp) +{ +CPUState *cpu; +unsigned long vcpu_id; +KVMState *s = kvm_state; +int kvm_fd, ret =0 ; + +CPU_FOREACH (cpu) { + vcpu_id =kvm_arch_vcpu_id (cpu); + +if (cpu->kvm_fd) { +close (cpu->kvm_fd); + } + + ret =kvm_arch_destroy_vcpu (cpu); +if (ret <0 ) { + goto err; + } + +if (s->coalesced_mmio_ring == (void *)cpu->kvm_run +PAGE_SIZE ) { + s->coalesced_mmio_ring =NULL ; + } + + ret =vcpu_unmap_regions (s, cpu); +if (ret <0 ) { + goto err; + } + + ret =kvm_arch_pre_create_vcpu (cpu, errp); +if (ret <0 ) { + goto err; + } + + kvm_fd =kvm_vm_ioctl (s,KVM_CREATE_VCPU , vcpu_id); +if (kvm_fd <0 ) { +error_report ("KVM_CREATE_VCPU IOCTL failed for vCPU %lu (%s)" , + vcpu_id,strerror (kvm_fd)); +return kvm_fd; + } + + cpu->kvm_fd = kvm_fd; + + cpu->vcpu_dirty = false; + cpu->dirty_pages =0 ; + cpu->throttle_us_per_full =0 ; + + ret =map_kvm_run (s, cpu, errp); +if (ret <0 ) { + goto err; + } + +if (s->kvm_dirty_ring_size) { + ret =map_kvm_dirty_gfns (s, cpu, errp); +if (ret <0 ) { + goto err; + } + } + + ret =kvm_arch_init_vcpu (cpu); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"kvm_init_vcpu: kvm_arch_init_vcpu failed (%lu)" , + vcpu_id); + } + +close (cpu->kvm_vcpu_stats_fd); + cpu->kvm_vcpu_stats_fd =kvm_vcpu_ioctl (cpu,KVM_GET_STATS_FD ,NULL ); +kvm_init_cpu_signals (cpu); + +kvm_cpu_synchronize_post_init (cpu); + } +trace_kvm_rebind_vcpus (); + + err: +return ret; +} +static void kvm_park_vcpu (CPUState *cpu) {struct KVMParkedVcpu *vcpu;
@@ -508,19 +589,11 @@ int kvm_create_and_park_vcpu(CPUState *cpu)return ret; } -static int do_kvm_destroy_vcpu (CPUState *cpu) +static int vcpu_unmap_regions (KVMState *s,CPUState *cpu) { -KVMState *s = kvm_state;int mmap_size;int ret =0 ; -trace_kvm_destroy_vcpu (cpu->cpu_index,kvm_arch_vcpu_id (cpu)); - - ret =kvm_arch_destroy_vcpu (cpu); -if (ret <0 ) { - goto err; - } - mmap_size =kvm_ioctl (s,KVM_GET_VCPU_MMAP_SIZE ,0 );if (mmap_size <0 ) { ret = mmap_size;
@@ -548,39 +621,47 @@ static int do_kvm_destroy_vcpu(CPUState *cpu) cpu->kvm_dirty_gfns =NULL ; } -kvm_park_vcpu (cpu); -err: + err:return ret; } -void kvm_destroy_vcpu (CPUState *cpu) -{ -if (do_kvm_destroy_vcpu (cpu) <0 ) { -error_report ("kvm_destroy_vcpu failed" ); -exit (EXIT_FAILURE ); - } -} - -int kvm_init_vcpu (CPUState *cpu,Error **errp) +static int do_kvm_destroy_vcpu (CPUState *cpu) {KVMState *s = kvm_state; -int mmap_size; -int ret; +int ret =0 ; -trace_kvm_init_vcpu (cpu->cpu_index,kvm_arch_vcpu_id (cpu)); +trace_kvm_destroy_vcpu (cpu->cpu_index,kvm_arch_vcpu_id (cpu)); - ret =kvm_arch_pre_create_vcpu (cpu, errp); + ret =kvm_arch_destroy_vcpu (cpu);if (ret <0 ) { goto err; } - ret =kvm_create_vcpu (cpu); +/* If I am the CPU that created coalesced_mmio_ring, then discard it */ +if (s->coalesced_mmio_ring == (void *)cpu->kvm_run +PAGE_SIZE ) { + s->coalesced_mmio_ring =NULL ; + } + + ret =vcpu_unmap_regions (s, cpu);if (ret <0 ) { -error_setg_errno (errp, -ret, -"kvm_init_vcpu: kvm_create_vcpu failed (%lu)" , -kvm_arch_vcpu_id (cpu)); goto err; } +kvm_park_vcpu (cpu); +err: +return ret; +} + +void kvm_destroy_vcpu (CPUState *cpu) +{ +if (do_kvm_destroy_vcpu (cpu) <0 ) { +error_report ("kvm_destroy_vcpu failed" ); +exit (EXIT_FAILURE ); + } +} + +static int map_kvm_run (KVMState *s,CPUState *cpu,Error **errp) +{ +int mmap_size, ret =0 ; mmap_size =kvm_ioctl (s,KVM_GET_VCPU_MMAP_SIZE ,0 );if (mmap_size <0 ) {
@@ -605,14 +686,53 @@ int kvm_init_vcpu(CPUState *cpu, Error **errp) (void *)cpu->kvm_run + s->coalesced_mmio *PAGE_SIZE ; } + err: +return ret; +} + +static int map_kvm_dirty_gfns (KVMState *s,CPUState *cpu,Error **errp) +{ +int ret =0 ; +/* Use MAP_SHARED to share pages with the kernel */ + cpu->kvm_dirty_gfns =mmap (NULL , s->kvm_dirty_ring_bytes, +PROT_READ |PROT_WRITE ,MAP_SHARED , + cpu->kvm_fd, +PAGE_SIZE *KVM_DIRTY_LOG_PAGE_OFFSET ); +if (cpu->kvm_dirty_gfns ==MAP_FAILED ) { + ret = -errno; + } + +return ret; +} + +int kvm_init_vcpu (CPUState *cpu,Error **errp) +{ +KVMState *s = kvm_state; +int ret; + +trace_kvm_init_vcpu (cpu->cpu_index,kvm_arch_vcpu_id (cpu)); + + ret =kvm_arch_pre_create_vcpu (cpu, errp); +if (ret <0 ) { + goto err; + } + + ret =kvm_create_vcpu (cpu); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"kvm_init_vcpu: kvm_create_vcpu failed (%lu)" , +kvm_arch_vcpu_id (cpu)); + goto err; + } + + ret =map_kvm_run (s, cpu, errp); +if (ret <0 ) { + goto err; + } +if (s->kvm_dirty_ring_size) { -/* Use MAP_SHARED to share pages with the kernel */ - cpu->kvm_dirty_gfns =mmap (NULL , s->kvm_dirty_ring_bytes, -PROT_READ |PROT_WRITE ,MAP_SHARED , - cpu->kvm_fd, -PAGE_SIZE *KVM_DIRTY_LOG_PAGE_OFFSET ); -if (cpu->kvm_dirty_gfns ==MAP_FAILED ) { - ret = -errno; + ret =map_kvm_dirty_gfns (s, cpu, errp); +if (ret <0 ) { goto err; } }
@@ -2710,6 +2830,16 @@ static int kvm_reset_vmfd(MachineState *ms) }assert (!err); +/* +* rebind new vcpu fds with the new kvm fds +* These can only be called after kvm_arch_vmfd_change_ops() +*/ + ret =kvm_rebind_vcpus (&err); +if (ret <0 ) { +return ret; + } +assert (!err); +/* these can be only called after ram_block_rebind() */ memory_listener_register (&kml->listener, &address_space_memory);memory_listener_register (&kvm_io_listener, &address_space_io);
accel/kvm/trace-events +1 -0
@@ -15,6 +15,7 @@ kvm_park_vcpu(int cpu_index, unsigned long arch_cpu_id) "index: %d id: %lu" kvm_unpark_vcpu(unsigned long arch_cpu_id, const char *msg) "id: %lu %s" kvm_irqchip_commit_routes(void) "" kvm_reset_vmfd(void) "" +kvm_rebind_vcpus(void) "" kvm_irqchip_add_msi_route(char *name, int vector, int virq) "dev %s vector %d virq %d" kvm_irqchip_update_msi_route(int virq) "Updating MSI route virq=%d" kvm_irqchip_release_virq(int virq) "virq %d"
Signature · 1 line
2.42.0
PATCH v3 15/33 i386/tdx: refactor TDX firmware memory initialization code into a new function
In reply to Ani SinhaA new helper function is introduced that refactors all firmware memory initialization code into a separate function. No functional change.
--- target/i386/kvm/tdx.c | 73 ++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 33 deletions(-)
target/i386/kvm/tdx.c +40 -33
@@ -295,14 +295,51 @@ static void tdx_post_init_vcpus(void) } } -static void tdx_finalize_vm (Notifier *notifier,void *unused) +static void tdx_init_fw_mem_region (void ) {TdxFirmware *tdvf = &tdx_guest->tdvf;TdxFirmwareEntry *entry; -RAMBlock *ram_block;Error *local_err =NULL ;int r; +for_each_tdx_fw_entry (tdvf, entry) { +struct kvm_tdx_init_mem_region region; +uint32_t flags; + + region = (struct kvm_tdx_init_mem_region ) { + .source_addr = (uintptr_t )entry->mem_ptr, + .gpa = entry->address, + .nr_pages = entry->size >>12 , + }; + + flags = entry->attributes &TDVF_SECTION_ATTRIBUTES_MR_EXTEND ? +KVM_TDX_MEASURE_MEMORY_REGION :0 ; + +do { +error_free (local_err); + local_err =NULL ; + r =tdx_vcpu_ioctl (first_cpu,KVM_TDX_INIT_MEM_REGION , flags, + ®ion, &local_err); + }while (r == -EAGAIN || r == -EINTR ); +if (r <0 ) { +error_report_err (local_err); +exit (1 ); + } + +if (entry->type ==TDVF_SECTION_TYPE_TD_HOB || + entry->type ==TDVF_SECTION_TYPE_TEMP_MEM ) { +qemu_ram_munmap (-1 , entry->mem_ptr, entry->size); + entry->mem_ptr =NULL ; + } + } +} + +static void tdx_finalize_vm (Notifier *notifier,void *unused) +{ +TdxFirmware *tdvf = &tdx_guest->tdvf; +TdxFirmwareEntry *entry; +RAMBlock *ram_block; +tdx_init_ram_entries ();for_each_tdx_fw_entry (tdvf, entry) {
@@ -339,37 +376,7 @@ static void tdx_finalize_vm(Notifier *notifier, void *unused)tdvf_hob_create (tdx_guest,tdx_get_hob_entry (tdx_guest));tdx_post_init_vcpus (); - -for_each_tdx_fw_entry (tdvf, entry) { -struct kvm_tdx_init_mem_region region; -uint32_t flags; - - region = (struct kvm_tdx_init_mem_region ) { - .source_addr = (uintptr_t )entry->mem_ptr, - .gpa = entry->address, - .nr_pages = entry->size >>12 , - }; - - flags = entry->attributes &TDVF_SECTION_ATTRIBUTES_MR_EXTEND ? -KVM_TDX_MEASURE_MEMORY_REGION :0 ; - -do { -error_free (local_err); - local_err =NULL ; - r =tdx_vcpu_ioctl (first_cpu,KVM_TDX_INIT_MEM_REGION , flags, - ®ion, &local_err); - }while (r == -EAGAIN || r == -EINTR ); -if (r <0 ) { -error_report_err (local_err); -exit (1 ); - } - -if (entry->type ==TDVF_SECTION_TYPE_TD_HOB || - entry->type ==TDVF_SECTION_TYPE_TEMP_MEM ) { -qemu_ram_munmap (-1 , entry->mem_ptr, entry->size); - entry->mem_ptr =NULL ; - } - } +tdx_init_fw_mem_region ();/* * TDVF image has been copied into private region above via
Signature · 1 line
2.42.0
PATCH v3 16/33 i386/tdx: finalize TDX guest state upon reset
In reply to Ani SinhaWhen the confidential virtual machine KVM file descriptor changes due to the guest reset, some TDX specific setup steps needs to be done again. This includes finalizing the inital guest launch state again. This change re-executes some parts of the TDX setup during the device reset phaze using a resettable interface. This finalizes the guest launch state again and locks it in. Machine done notifier which was previously used is no longer needed as the same code is now executed as a part of VM reset.
--- target/i386/kvm/tdx.c | 38 +++++++++++++++++++++++++++++++----- target/i386/kvm/tdx.h | 1 + target/i386/kvm/trace-events | 3 +++ 3 files changed, 37 insertions(+), 5 deletions(-)
target/i386/kvm/tdx.c +33 -5
@@ -19,6 +19,7 @@#include "crypto/hash.h" #include "system/kvm_int.h" #include "system/runstate.h" +#include "system/reset.h" #include "system/system.h" #include "system/ramblock.h" #include "system/address-spaces.h"
@@ -38,6 +39,7 @@#include "kvm_i386.h" #include "tdx.h" #include "tdx-quote-generator.h" +#include "trace.h" #include "standard-headers/asm-x86/kvm_para.h"
@@ -389,9 +391,19 @@ static void tdx_finalize_vm(Notifier *notifier, void *unused)CONFIDENTIAL_GUEST_SUPPORT (tdx_guest)->ready = true; } -static Notifier tdx_machine_done_notify = { - .notify = tdx_finalize_vm, -}; +static void tdx_handle_reset (Object *obj,ResetType type) +{ +if (!runstate_is_running () && !phase_check (PHASE_MACHINE_READY )) { +return ; + } + +if (!kvm_enable_hypercall (BIT_ULL (KVM_HC_MAP_GPA_RANGE ))) { +error_setg (&error_fatal,"KVM_HC_MAP_GPA_RANGE not enabled for guest" ); + } + +tdx_finalize_vm (NULL ,NULL ); +trace_tdx_handle_reset (); +}/* * Some CPUID bits change from fixed1 to configurable bits when TDX module
@@ -738,8 +750,6 @@ static int tdx_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)*/ kvm_readonly_mem_allowed = false; -qemu_add_machine_init_done_notifier (&tdx_machine_done_notify); - tdx_guest = tdx;return 0 ; }
@@ -1505,6 +1515,7 @@ OBJECT_DEFINE_TYPE_WITH_INTERFACES(TdxGuest,TDX_GUEST ,X86_CONFIDENTIAL_GUEST , { TYPE_USER_CREATABLE }, + { TYPE_RESETTABLE_INTERFACE }, { NULL }) static void tdx_guest_init(Object *obj)
@@ -1538,16 +1549,24 @@ static void tdx_guest_init(Object *obj) tdx->event_notify_vector =-1 ; tdx->event_notify_apicid =-1 ; +qemu_register_resettable (obj); }static void tdx_guest_finalize (Object *obj) { } +static ResettableState *tdx_reset_state (Object *obj) +{ +TdxGuest *tdx =TDX_GUEST (obj); +return &tdx->reset_state; +} +static void tdx_guest_class_init (ObjectClass *oc,const void *data) {ConfidentialGuestSupportClass *klass =CONFIDENTIAL_GUEST_SUPPORT_CLASS (oc);X86ConfidentialGuestClass *x86_klass =X86_CONFIDENTIAL_GUEST_CLASS (oc); +ResettableClass *rc =RESETTABLE_CLASS (oc); klass->kvm_init = tdx_kvm_init; klass->can_rebuild_guest_state = true;
@@ -1555,4 +1574,13 @@ static void tdx_guest_class_init(ObjectClass *oc, const void *data) x86_klass->cpu_instance_init = tdx_cpu_instance_init; x86_klass->adjust_cpuid_features = tdx_adjust_cpuid_features; x86_klass->check_features = tdx_check_features; + +/* +* the exit phase makes sure sev handles reset after all legacy resets +* have taken place (in the hold phase) and IGVM has also properly +* set up the boot state. +*/ + rc->phases.exit = tdx_handle_reset; + rc->get_state = tdx_reset_state; + }
target/i386/kvm/tdx.h +1 -0
@@ -70,6 +70,7 @@ typedef struct TdxGuest {uint32_t event_notify_vector;uint32_t event_notify_apicid; +ResettableState reset_state; } TdxGuest;#ifdef CONFIG_TDX
target/i386/kvm/trace-events +3 -0
@@ -14,3 +14,6 @@ kvm_xen_soft_reset(void) "" kvm_xen_set_shared_info(uint64_t gfn) "shared info at gfn 0x%" PRIx64 kvm_xen_set_vcpu_attr(int cpu, int type, uint64_t gpa) "vcpu attr cpu %d type %d gpa 0x%" PRIx64 kvm_xen_set_vcpu_callback(int cpu, int vector) "callback vcpu %d vector %d" + +# tdx.c +tdx_handle_reset(void) ""
Signature · 1 line
2.42.0
PATCH v3 17/33 i386/tdx: add a pre-vmfd change notifier to reset tdx state
In reply to Ani SinhaDuring reset, when the VM file descriptor is changed, the TDX state needs to be re-initialized. A notifier callback is implemented to reset the old state and free memory before the new state is initialized post VM file descriptor change.
--- target/i386/kvm/tdx.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
target/i386/kvm/tdx.c +31 -0
@@ -405,6 +405,36 @@ static void tdx_handle_reset(Object *obj, ResetType type)trace_tdx_handle_reset (); } +/* TDX guest reset will require us to reinitialize some of tdx guest state. */ +static int set_tdx_vm_uninitialized (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +TdxFirmware *fw = &tdx_guest->tdvf; + +if (!((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (tdx_guest->initialized) { + tdx_guest->initialized = false; + } + +g_free (tdx_guest->ram_entries); + +/* +* the firmware entries will be parsed again, see +* x86_firmware_configure() -> tdx_parse_tdvf() +*/ + fw->entries =0 ; +g_free (fw->entries); + +return 0 ; +} + +static NotifierWithReturn tdx_vmfd_change_notifier = { + .notify = set_tdx_vm_uninitialized, +}; +/* * Some CPUID bits change from fixed1 to configurable bits when TDX module * supports TDX_FEATURES0.VE_REDUCTION. e.g., MCA/MCE/MTRR/CORE_CAPABILITY.
@@ -1549,6 +1579,7 @@ static void tdx_guest_init(Object *obj) tdx->event_notify_vector =-1 ; tdx->event_notify_apicid =-1 ; +kvm_vmfd_add_change_notifier (&tdx_vmfd_change_notifier);qemu_register_resettable (obj); }
Signature · 1 line
2.42.0
PATCH v3 18/33 i386/sev: add migration blockers only once
In reply to Ani Sinhasev_launch_finish() and sev_snp_launch_finish() could be called multiple times when the confidential guest is being reset/rebooted. The migration blockers should not be added multiple times, once per invocation. This change makes sure that the migration blockers are added only one time by adding the migration blockers to the vm state change handler when the vm transitions to the running state. Subsequent reboots do not change the state of the vm.
--- target/i386/sev.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-)
target/i386/sev.c +5 -15
@@ -1420,11 +1420,6 @@ sev_launch_finish(SevCommonState *sev_common) }sev_set_guest_state (sev_common,SEV_STATE_RUNNING ); - -/* add migration blocker */ -error_setg (&sev_mig_blocker, -"SEV: Migration is not implemented" ); -migrate_add_blocker (&sev_mig_blocker, &error_fatal); } static int snp_launch_update_data(uint64_t gpa, void *hva, size_t len,
@@ -1607,7 +1602,6 @@ static void sev_snp_launch_finish(SevCommonState *sev_common) {int ret, error; -Error *local_err =NULL ;OvmfSevMetadata *metadata;SevLaunchUpdateData *data;SevSnpGuestState *sev_snp =SEV_SNP_GUEST (sev_common);
@@ -1654,15 +1648,6 @@ sev_snp_launch_finish(SevCommonState *sev_common)kvm_mark_guest_state_protected ();sev_set_guest_state (sev_common,SEV_STATE_RUNNING ); - -/* add migration blocker */ -error_setg (&sev_mig_blocker, -"SEV-SNP: Migration is not implemented" ); - ret =migrate_add_blocker (&sev_mig_blocker, &local_err); -if (local_err) { -error_report_err (local_err); -exit (1 ); - } }
@@ -1675,6 +1660,11 @@ sev_vm_state_change(void *opaque, bool running, RunState state)if (running) {if (!sev_check_state (sev_common,SEV_STATE_RUNNING )) { klass->launch_finish (sev_common); + +/* add migration blocker */ +error_setg (&sev_mig_blocker, +"SEV: Migration is not implemented" ); +migrate_add_blocker (&sev_mig_blocker, &error_fatal); } } }
Signature · 1 line
2.42.0
PATCH v3 19/33 i386/sev: add notifiers only once
In reply to Ani SinhaThe various notifiers that are used needs to be installed only once not on every initialization. This includes the vm state change notifier and others. This change uses 'cgs->ready' flag to install the notifiers only one time, the first time.
--- target/i386/sev.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-)
target/i386/sev.c +19 -17
@@ -1920,8 +1920,9 @@ static int sev_common_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)return -1 ; } -qemu_add_vm_change_state_handler (sev_vm_state_change, sev_common); - +if (!cgs->ready) { +qemu_add_vm_change_state_handler (sev_vm_state_change, sev_common); + } cgs->ready = true;return 0 ;
@@ -1943,22 +1944,23 @@ static int sev_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)return -1 ; } -/* -* SEV uses these notifiers to register/pin pages prior to guest use, -* but SNP relies on guest_memfd for private pages, which has its -* own internal mechanisms for registering/pinning private memory. -*/ -ram_block_notifier_add (&sev_ram_notifier); - -/* -* The machine done notify event is used for SEV guests to get the -* measurement of the encrypted images. When SEV-SNP is enabled, the -* measurement is part of the guest attestation process where it can -* be collected without any reliance on the VMM. So skip registering -* the notifier for SNP in favor of using guest attestation instead. -*/ -qemu_add_machine_init_done_notifier (&sev_machine_done_notify); +if (!cgs->ready) { +/* +* SEV uses these notifiers to register/pin pages prior to guest use, +* but SNP relies on guest_memfd for private pages, which has its +* own internal mechanisms for registering/pinning private memory. +*/ +ram_block_notifier_add (&sev_ram_notifier); +/* +* The machine done notify event is used for SEV guests to get the +* measurement of the encrypted images. When SEV-SNP is enabled, the +* measurement is part of the guest attestation process where it can +* be collected without any reliance on the VMM. So skip registering +* the notifier for SNP in favor of using guest attestation instead. +*/ +qemu_add_machine_init_done_notifier (&sev_machine_done_notify); + }return 0 ; }
Signature · 1 line
2.42.0
PATCH v3 20/33 i386/sev: free existing launch update data and kernel hashes data on init
In reply to Ani SinhaIf there is existing launch update data and kernel hashes data, they need to be freed when initialization code is executed. This is important for resettable confidential guests where the initialization happens once every reset.
--- target/i386/sev.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
target/i386/sev.c +12 -0
@@ -1772,6 +1772,7 @@ static int sev_common_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)uint32_t ebx;uint32_t host_cbitpos;struct sev_user_data_status status = {}; +SevLaunchUpdateData *data, *next_elm;SevCommonState *sev_common =SEV_COMMON (cgs);SevCommonStateClass *klass =SEV_COMMON_GET_CLASS (cgs); X86ConfidentialGuestClass *x86_klass =
@@ -1779,6 +1780,11 @@ static int sev_common_kvm_init(ConfidentialGuestSupport *cgs, Error **errp) sev_common->state =SEV_STATE_UNINIT ; +/* free existing launch update data if any */ + QTAILQ_FOREACH_SAFE(data, &launch_update, next, next_elm) { +g_free (data); + } +host_cpuid (0x8000001F ,0 ,NULL , &ebx,NULL ,NULL ); host_cbitpos = ebx &0x3f ;
@@ -1968,6 +1974,8 @@ static int sev_snp_kvm_init(ConfidentialGuestSupport *cgs, Error **errp) {MachineState *ms =MACHINE (qdev_get_machine ());X86MachineState *x86ms =X86_MACHINE (ms); +SevCommonState *sev_common =SEV_COMMON (cgs); +SevSnpGuestState *sev_snp_guest =SEV_SNP_GUEST (sev_common);if (x86ms->smm ==ON_OFF_AUTO_AUTO ) { x86ms->smm =ON_OFF_AUTO_OFF ;
@@ -1976,6 +1984,10 @@ static int sev_snp_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)return -1 ; } +/* free existing kernel hashes data if any */ +g_free (sev_snp_guest->kernel_hashes_data); + sev_snp_guest->kernel_hashes_data =NULL ; +return 0 ; }
Signature · 1 line
2.42.0
PATCH v3 21/33 i386/sev: add support for confidential guest reset
In reply to Ani SinhaWhen the KVM VM file descriptor changes as a part of the confidential guest reset mechanism, it necessary to create a new confidential guest context and re-encrypt the VM memeory. This happens for SEV-ES and SEV-SNP virtual machines as a part of SEV_LAUNCH_FINISH, SEV_SNP_LAUNCH_FINISH operations.
A new resettable interface for SEV module has been added. A new reset callback for the reset 'exit' state has been implemented to perform the above operations when the VM file descriptor has changed during VM reset.
Tracepoints has been added also for tracing purpose.
--- target/i386/sev.c | 58 ++++++++++++++++++++++++++++++++++++++++ target/i386/trace-events | 1 + 2 files changed, 59 insertions(+)
target/i386/sev.c +58 -0
@@ -30,8 +30,10 @@#include "system/kvm.h" #include "kvm/kvm_i386.h" #include "sev.h" +#include "system/cpus.h" #include "system/system.h" #include "system/runstate.h" +#include "system/reset.h" #include "trace.h" #include "migration/blocker.h" #include "qom/object.h"
@@ -85,6 +87,10 @@ typedef struct QEMU_PACKED PaddedSevHashTable {uint8_t padding[ROUND_UP (sizeof (SevHashTable),16 ) -sizeof (SevHashTable)]; } PaddedSevHashTable; +static void sev_handle_reset (Object *obj,ResetType type); + +SevKernelLoaderContext sev_load_ctx = {}; +QEMU_BUILD_BUG_ON (sizeof (PaddedSevHashTable) %16 !=0 );#define SEV_INFO_BLOCK_GUID "00f771de-1a7e-4fcb-890e-68c77e2fb44e"
@@ -128,6 +134,7 @@ struct SevCommonState {uint8_t build_id;int sev_fd;SevState state; +ResettableState reset_state; QTAILQ_HEAD(, SevLaunchVmsa) launch_vmsa; };
@@ -1665,6 +1672,11 @@ sev_vm_state_change(void *opaque, bool running, RunState state)error_setg (&sev_mig_blocker,"SEV: Migration is not implemented" );migrate_add_blocker (&sev_mig_blocker, &error_fatal); +/* +* mark SEV guest as resettable so that we can reinitialize +* SEV upon reset. +*/ +qemu_register_resettable (OBJECT (sev_common)); } } }
@@ -1991,6 +2003,41 @@ static int sev_snp_kvm_init(ConfidentialGuestSupport *cgs, Error **errp)return 0 ; } +/* +* handle sev vm reset +*/ +static void sev_handle_reset (Object *obj,ResetType type) +{ +SevCommonState *sev_common =SEV_COMMON (MACHINE (qdev_get_machine ())->cgs); +SevCommonStateClass *klass =SEV_COMMON_GET_CLASS (sev_common); + +if (!sev_common) { +return ; + } + +if (!runstate_is_running ()) { +return ; + } + +sev_add_kernel_loader_hashes (&sev_load_ctx, &error_fatal); +if (sev_es_enabled () && !sev_snp_enabled ()) { +sev_launch_get_measure (NULL ,NULL ); + } +if (!sev_check_state (sev_common,SEV_STATE_RUNNING )) { +/* this calls sev_snp_launch_finish() etc */ + klass->launch_finish (sev_common); + } + +trace_sev_handle_reset (); +return ; +} + +static ResettableState *sev_reset_state (Object *obj) +{ +SevCommonState *sev_common =SEV_COMMON (obj); +return &sev_common->reset_state; +} +int sev_encrypt_flash (hwaddr gpa,uint8_t *ptr,uint64_t len,Error **errp) {
@@ -2469,6 +2516,8 @@ bool sev_add_kernel_loader_hashes(SevKernelLoaderContext *ctx, Error **errp)return false; } +/* save the context here so that it can be re-used when vm is reset */ +memcpy (&sev_load_ctx, ctx,sizeof (*ctx));return klass->build_kernel_loader_hashes (sev_common, area, ctx, errp); }
@@ -2729,8 +2778,16 @@ static void sev_common_class_init(ObjectClass *oc, const void *data) {ConfidentialGuestSupportClass *klass =CONFIDENTIAL_GUEST_SUPPORT_CLASS (oc); +ResettableClass *rc =RESETTABLE_CLASS (oc); klass->kvm_init = sev_common_kvm_init; +/* +* the exit phase makes sure sev handles reset after all legacy resets +* have taken place (in the hold phase) and IGVM has also properly +* set up the boot state. +*/ + rc->phases.exit = sev_handle_reset; + rc->get_state = sev_reset_state; object_class_property_add_str(oc, "sev-device", sev_common_get_sev_device,
@@ -2780,6 +2837,7 @@ static const TypeInfo sev_common_info = { .abstract = true, .interfaces = (const InterfaceInfo []) { {TYPE_USER_CREATABLE }, + {TYPE_RESETTABLE_INTERFACE }, { } } };
target/i386/trace-events +1 -0
@@ -14,3 +14,4 @@ kvm_sev_attestation_report(const char *mnonce, const char *data) "mnonce %s data kvm_sev_snp_launch_start(uint64_t policy, char *gosvw) "policy 0x%" PRIx64 " gosvw %s" kvm_sev_snp_launch_update(uint64_t src, uint64_t gpa, uint64_t len, const char *type) "src 0x%" PRIx64 " gpa 0x%" PRIx64 " len 0x%" PRIx64 " (%s page)" kvm_sev_snp_launch_finish(char *id_block, char *id_auth, char *host_data) "id_block %s id_auth %s host_data %s" +sev_handle_reset(void) ""
Signature · 1 line
2.42.0
PATCH v3 22/33 hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
In reply to Ani SinhaNormally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)
hw/vfio/helpers.c +83 -3
@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )
@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;
@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {
@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }
@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );
@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; }
Signature · 1 line
2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Otherwise I am going to add a comment in the next spin up that this change is not tested.
Quoted 144 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +83
-4@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; } --2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 10 lines · Ani Sinha
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Automated VFIO testing [*] is tough because it depends on a "real" physical PCI device.
I have tested you series using a sev-snp guest with an assigned device (SATA controller) and verified that the guest could reboot. The SATA controller has no disks connected, so further testing is required.
However, a 'system_reset' command issued from the QEMU monitor leaves the guest hanging. May be this is expected ?
Quoted 2 lines
Otherwise I am going to add a comment in the next spin up that this change is not tested.
I will provide some more comments on the code. Please give me a day or two before resending.
Thanks,
C.
[*] One way to address this constraint for VFIO testing would be
to run the test in a nested environment using emulated devices, like IGB, attached to an L1 which would be assigned to an L2. But that's unrelated to this series.
Quoted 146 lines
Quoted 145 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +83
-4@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; } --2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 16 lines · Cédric Le Goater
Quoted 10 lines · Ani Sinha
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Automated VFIO testing [*] is tough because it depends on a "real" physical PCI device.
Quoted 6 lines
I have tested you series using a sev-snp guest with an assigned device (SATA controller) and verified that the guest could reboot. The SATA controller has no disks connected, so further testing is required.
However, a '
system_reset' command issued from the QEMU monitor leaves the guest hanging. May be this is expected ?
I do not think so. On non-coco, I just tried it and it reboots fine with x-change-vmfd-on-reset turned on.
I tried a RHEL 10 guest image as well as a simple UKI kernel. Maybe there is some bug in this patch, not sure. In QEMU the code path should be the same for both.
Quoted 7 lines
Quoted 4 lines
Otherwise I am going to add a comment in the next spin up that this change is not tested.
I will provide some more comments on the code. Please give me a day or two before resending.
Sure, please take all the time you need.
Quoted 156 lines
Thanks,
C.
[*] One way to address this constraint for VFIO testing would be
to run the test in a nested environment using emulated devices, like IGB, attached to an L1 which would be assigned to an L2. But that's unrelated to this series.Quoted 146 lines
Quoted 145 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +83
-4@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; } --2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 22 lines · Ani Sinha
Quoted 20 lines · Cédric Le Goater
Quoted 10 lines · Ani Sinha
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Automated VFIO testing [*] is tough because it depends on a "real" physical PCI device.
I have tested you series using a sev-snp guest with an assigned device (SATA controller) and verified that the guest could reboot. The SATA controller has no disks connected, so further testing is required.
Quoted 5 lines
Quoted 2 lines
However, a '
system_reset' command issued from the QEMU monitor leaves the guest hanging. May be this is expected ?I do not think so. On non-coco, I just tried it and it reboots fine with x-change-vmfd-on-reset turned on.
Please try this way on non-coco. If it works, then we need to assume that for the CoCo case, some extra steps are needed during reset for vfio which I missed.
Quoted 170 lines
I tried a RHEL 10 guest image as well as a simple UKI kernel. Maybe there is some bug in this patch, not sure. In QEMU the code path should be the same for both.
Quoted 7 lines
Quoted 4 lines
Otherwise I am going to add a comment in the next spin up that this change is not tested.
I will provide some more comments on the code. Please give me a day or two before resending.
Sure, please take all the time you need.
Quoted 156 lines
Thanks,
C.
[*] One way to address this constraint for VFIO testing would be
to run the test in a nested environment using emulated devices, like IGB, attached to an L1 which would be assigned to an L2. But that's unrelated to this series.Quoted 146 lines
Quoted 145 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +83
-4@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; } --2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
In reply to Ani SinhaQuoted 25 lines · Ani Sinha
Quoted 20 lines · Cédric Le Goater
Quoted 10 lines · Ani Sinha
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Automated VFIO testing [*] is tough because it depends on a "real" physical PCI device.
I have tested you series using a sev-snp guest with an assigned device (SATA controller) and verified that the guest could reboot. The SATA controller has no disks connected, so further testing is required.
Quoted 5 lines
Quoted 2 lines
However, a '
system_reset' command issued from the QEMU monitor leaves the guest hanging. May be this is expected ?I do not think so. On non-coco, I just tried it and it reboots fine with x-change-vmfd-on-reset turned on.
Sorry I had some issues with my testing. Neither non-coco reset not coco reset was using vmfd change path when the QMP reset command was used. In non-coco, it was the normal reset which obviously works.
I fixed it. See https://gitlab.com/anisinha/qemu/-/commit/c5514dbb2b52d474397220cfa3a0b8faa26cda0c
Please use this patch and test. I have tasted this on SNP and seems to work. I will make it part of next spin-up.
Thanks for testing.
Quoted 170 lines
I tried a RHEL 10 guest image as well as a simple UKI kernel. Maybe there is some bug in this patch, not sure. In QEMU the code path should be the same for both.
Quoted 7 lines
Quoted 4 lines
Otherwise I am going to add a comment in the next spin up that this change is not tested.
I will provide some more comments on the code. Please give me a day or two before resending.
Sure, please take all the time you need.
Quoted 156 lines
Thanks,
C.
[*] One way to address this constraint for VFIO testing would be
to run the test in a nested environment using emulated devices, like IGB, attached to an L1 which would be assigned to an L2. But that's unrelated to this series.Quoted 146 lines
Quoted 145 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +83
-4@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* * We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and * we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ; + +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp); +static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) { +close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd); return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + } +#endif return 0 ; } --2.42.0
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 31 lines · Ani Sinha
Quoted 25 lines · Ani Sinha
Quoted 20 lines · Cédric Le Goater
Quoted 10 lines · Ani Sinha
Quoted 6 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
I have not been able to test this change. Can someone suggest a way to add a functional test and/or test this manually?
Automated VFIO testing [*] is tough because it depends on a "real" physical PCI device.
I have tested you series using a sev-snp guest with an assigned device (SATA controller) and verified that the guest could reboot. The SATA controller has no disks connected, so further testing is required.
Quoted 5 lines
Quoted 2 lines
However, a '
system_reset' command issued from the QEMU monitor leaves the guest hanging. May be this is expected ?I do not think so. On non-coco, I just tried it and it reboots fine with x-change-vmfd-on-reset turned on.
Quoted 6 lines
Sorry I had some issues with my testing. Neither non-coco reset not coco reset was using vmfd change path when the QMP reset command was used. In non-coco, it was the normal reset which obviously works.
I fixed it. See https://gitlab.com/anisinha/qemu/-/commit/c5514dbb2b52d474397220cfa3a0b8faa26cda0c
Please use this patch and test. I have tasted this on SNP and seems to work. I will make it part of next spin-up.
Looks to me.
Thanks,
C.
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
In reply to Ani SinhaQuoted 18 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +0
-0@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /*
hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 118
Quoted 10 lines
hw/vfio/helpers.c +6
-3* We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and
Instead of changing this paragraph, I would prefer having, below the vfio_kvm_device_fd definition, a new section starting with comment "Confidential virtual machines"
hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 121
Quoted 3 lines
hw/vfio/helpers.c +0
-0* we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ;
/*
* Confidential virtual machines
<what is different from non confidential VMs>
*/
<Code>
hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 132
Quoted 11 lines
hw/vfio/helpers.c +11
-0+ +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp);
This declaration could be avoided.
Quoted 18 lines
hw/vfio/helpers.c +18
-0+static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */
hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 160
Quoted 10 lines
hw/vfio/helpers.c +10
-0+if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) {
May be "vfio_kvm_device_fd != -1" ?
Quoted 41 lines
hw/vfio/helpers.c +23
-0+close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {
hw/vfio/helpers.c int vfio_kvm_device_add_fd(int fd, Error **errp) · line 232
Quoted 8 lines
hw/vfio/helpers.c +5
-0@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd);return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +
Please introduce a helper routine.
Quoted 13 lines
hw/vfio/helpers.c +1
-0#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);
hw/vfio/helpers.c int vfio_kvm_device_del_fd(int fd, Error **errp) · line 264
Quoted 10 lines
hw/vfio/helpers.c +8
-0return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { +if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + }
and a second helper for removal.
Thanks,
C.
Quoted 3 lines
hw/vfio/helpers.c +0
-0#endif return 0 ; }
Re: [PATCH v3 22/33] hw/vfio: generate new file fd for pseudo device and rebind existing descriptors
Quoted 142 lines · Cédric Le Goater
Quoted 16 lines · Ani Sinha
Normally the vfio pseudo device file descriptor lives for the life of the VM. However, when the kvm VM file descriptor changes, a new file descriptor for the pseudo device needs to be generated against the new kvm VM descriptor. Other existing vfio descriptors needs to be reattached to the new pseudo device descriptor. This change performs the above steps.
Signed-off-by Ani Sinha <[email protected]>--- hw/vfio/helpers.c | 86 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 3 deletions(-)hw/vfio/helpers.c +0
-0@@ -110,12 +110,71 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,#ifdef CONFIG_KVM /* hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 118
Quoted 10 lines
hw/vfio/helpers.c +6
-3* We have a single VFIO pseudo device per KVM VM. Once created it lives -* for the life of the VM. Closing the file descriptor only drops our -* reference to it and the device's reference to kvm. Therefore once -* initialized, this file descriptor is only released on QEMU exit and +* for the life of the VM except when the vm file descriptor changes for +* confidential virtual machines. In that case, the old file descriptor is +* closed and a new file descriptor is recreated. Closing the file descriptor +* only drops our reference to it and the device's reference to kvm. +* Therefore once initialized, this file descriptor is normally only released +* on QEMU exit (except for confidential VMs as stated above) and Instead of changing this paragraph, I would prefer having, below the
vfio_kvm_device_fddefinition, a new section starting with comment "Confidential virtual machines"hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 121
Quoted 3 lines
hw/vfio/helpers.c +0
-0* we'll re-use it should another vfio device be attached before then. */ int vfio_kvm_device_fd =-1 ;/*
* Confidential virtual machines<what is different from non confidential VMs>
*/
<Code>hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 132
Quoted 11 lines
hw/vfio/helpers.c +11
-0+ +typedef struct KVMVfioFileFd { +int fd; +QLIST_ENTRY (KVMVfioFileFd) node; +}KVMVfioFileFd ; + +static QLIST_HEAD(, KVMVfioFileFd) kvm_vfio_file_fds = +QLIST_HEAD_INITIALIZER (kvm_vfio_file_fds); + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp);This declaration could be avoided.
Quoted 18 lines
hw/vfio/helpers.c +18
-0+static struct NotifierWithReturn kvm_vfio_vmfd_change_notifier = { + .notify = kvm_vfio_filefd_rebind, +}; + +static int kvm_vfio_filefd_rebind (NotifierWithReturn *notifier,void *data, +Error **errp) +{ +KVMVfioFileFd *file_fd; +int ret =0 ; +struct kvm_device_attr attr = { + .group =KVM_DEV_VFIO_FILE , + .attr =KVM_DEV_VFIO_FILE_ADD , + }; +struct kvm_create_device cd = { + .type =KVM_DEV_TYPE_VFIO , + }; + +/* we are not interested in pre vmfd change notification */ hw/vfio/helpers.c bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, · line 160
Quoted 10 lines
hw/vfio/helpers.c +10
-0+if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +if (kvm_vm_ioctl (kvm_state,KVM_CREATE_DEVICE , &cd)) { +error_setg_errno (errp, errno,"Failed to create KVM VFIO device" ); +return -errno; + } + +if (vfio_kvm_device_fd) {May be "
vfio_kvm_device_fd!= -1" ?Quoted 37 lines
hw/vfio/helpers.c +23
-0+close (vfio_kvm_device_fd); + } + + vfio_kvm_device_fd = cd.fd; + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) { + attr.addr = (uint64_t )(unsigned long )&file_fd->fd; +if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) { +error_setg_errno (errp, errno, +"Failed to add fd %d to KVM VFIO device" , + file_fd->fd); + ret = -errno; + } + } +return ret; +} +#endif void vfio_kvm_device_close (void )@@ -137,6 +196,7 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_ADD, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (!kvm_enabled ()) {return 0 ;@@ -153,6 +213,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) } vfio_kvm_device_fd = cd.fd; +/* +* If the vm file descriptor changes, add a notifier so that we can +* re-create the vfio_kvm_device_fd. +*/ +kvm_vmfd_add_change_notifier (&kvm_vfio_vmfd_change_notifier); }if (ioctl (vfio_kvm_device_fd,KVM_SET_DEVICE_ATTR , &attr)) {hw/vfio/helpers.c int vfio_kvm_device_add_fd(int fd, Error **errp) · line 232
Quoted 8 lines
hw/vfio/helpers.c +5
-0@@ -160,6 +225,11 @@ int vfio_kvm_device_add_fd(int fd, Error **errp) fd);return -errno; } + + file_fd =g_malloc0 (sizeof (*file_fd)); + file_fd->fd = fd; +QLIST_INSERT_HEAD (&kvm_vfio_file_fds, file_fd, node); +Please introduce a helper routine.
Quoted 16 lines
Quoted 11 lines
hw/vfio/helpers.c +1
-0#endif return 0 ; }@@ -172,6 +242,7 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) .attr = KVM_DEV_VFIO_FILE_DEL, .addr = (uint64_t)(unsigned long)&fd, }; +KVMVfioFileFd *file_fd;if (vfio_kvm_device_fd <0 ) {error_setg (errp,"KVM VFIO device isn't created yet" );hw/vfio/helpers.c int vfio_kvm_device_del_fd(int fd, Error **errp) · line 258
Quoted 5 lines
hw/vfio/helpers.c +2
-0@@ -183,6 +254,15 @@ int vfio_kvm_device_del_fd(int fd, Error **errp) "Failed to remove fd %d from KVM VFIO device", fd);return -errno; } + + QLIST_FOREACH(file_fd, &kvm_vfio_file_fds, node) {
^^^^^^^^^^^^^
I believe this should be QLIST_FOREACH_SAFE . Will fix in next version.
Quoted 16 lines
hw/vfio/helpers.c int vfio_kvm_device_del_fd(int fd, Error **errp) · line 264
Quoted 6 lines
hw/vfio/helpers.c +6
-0+if (file_fd->fd == fd) { +QLIST_REMOVE (file_fd, node); +g_free (file_fd); +break ; + } + }and a second helper for removal.
Thanks,
C.
Quoted 3 lines
hw/vfio/helpers.c +0
-0#endif return 0 ; }
PATCH v3 23/33 kvm/i8254: refactor pit initialization into a helper
In reply to Ani SinhaThe initialization code will be used again by VM file descriptor change notifier callback in a subsequent change. So refactor common code into a new helper function.
--- hw/i386/kvm/i8254.c | 68 +++++++++++++++++++++++++-------------------- 1 file changed, 38 insertions(+), 30 deletions(-)
hw/i386/kvm/i8254.c +38 -30
@@ -60,6 +60,43 @@ struct KVMPITClass {DeviceRealize parent_realize; }; +static void do_pit_initialize (KVMPITState *s,Error **errp) +{ +struct kvm_pit_config config = { + .flags =0 , + }; +int ret; + + ret =kvm_vm_ioctl (kvm_state,KVM_CREATE_PIT2 , &config); +if (ret <0 ) { +error_setg (errp,"Create kernel PIC irqchip failed: %s" , +strerror (-ret)); +return ; + } +switch (s->lost_tick_policy) { +case LOST_TICK_POLICY_DELAY : +break ;/* enabled by default */ +case LOST_TICK_POLICY_DISCARD : +if (kvm_check_extension (kvm_state,KVM_CAP_REINJECT_CONTROL )) { +struct kvm_reinject_control control = { .pit_reinject =0 }; + + ret =kvm_vm_ioctl (kvm_state,KVM_REINJECT_CONTROL , &control); +if (ret <0 ) { +error_setg (errp, +"Can't disable in-kernel PIT reinjection: %s" , +strerror (-ret)); +return ; + } + } +break ; +default : +error_setg (errp,"Lost tick policy not supported." ); +return ; + } + +return ; +} +static void kvm_pit_update_clock_offset (KVMPITState *s) {int64_t offset, clock_offset;
@@ -241,42 +278,13 @@ static void kvm_pit_realizefn(DeviceState *dev, Error **errp)PITCommonState *pit =PIT_COMMON (dev);KVMPITClass *kpc =KVM_PIT_GET_CLASS (dev);KVMPITState *s =KVM_PIT (pit); -struct kvm_pit_config config = { - .flags =0 , - }; -int ret;if (!kvm_check_extension (kvm_state,KVM_CAP_PIT_STATE2 ) || !kvm_check_extension (kvm_state,KVM_CAP_PIT2 )) {error_setg (errp,"In-kernel PIT not available" ); } - ret =kvm_vm_ioctl (kvm_state,KVM_CREATE_PIT2 , &config); -if (ret <0 ) { -error_setg (errp,"Create kernel PIC irqchip failed: %s" , -strerror (-ret)); -return ; - } -switch (s->lost_tick_policy) { -case LOST_TICK_POLICY_DELAY : -break ;/* enabled by default */ -case LOST_TICK_POLICY_DISCARD : -if (kvm_check_extension (kvm_state,KVM_CAP_REINJECT_CONTROL )) { -struct kvm_reinject_control control = { .pit_reinject =0 }; - - ret =kvm_vm_ioctl (kvm_state,KVM_REINJECT_CONTROL , &control); -if (ret <0 ) { -error_setg (errp, -"Can't disable in-kernel PIT reinjection: %s" , -strerror (-ret)); -return ; - } - } -break ; -default : -error_setg (errp,"Lost tick policy not supported." ); -return ; - } +do_pit_initialize (s, errp);memory_region_init_io (&pit->ioports,OBJECT (dev),NULL ,NULL ,"kvm-pit" ,4 );
Signature · 1 line
2.42.0
PATCH v3 24/33 kvm/i8254: add support for confidential guest reset
In reply to Ani SinhaA confidential guest reset involves closing the old virtual machine KVM file descriptor and opening a new one. Since its a new KVM fd, PIT needs to be reinitialized again. This is done with the help of a notifier which is invoked upon KVM vm file desciptor change during confidential guest reset process.
--- hw/i386/kvm/i8254.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
hw/i386/kvm/i8254.c +20 -0
@@ -52,6 +52,8 @@ struct KVMPITState {LostTickPolicy lost_tick_policy;bool vm_stopped;int64_t kernel_clock_offset; + +NotifierWithReturn kvmpit_vmfd_change_notifier; };struct KVMPITClass {
@@ -203,6 +205,21 @@ static void kvm_pit_put(PITCommonState *pit) } } +static int kvmpit_post_vmfd_change (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +KVMPITState *s =container_of (notifier, KVMPITState, + kvmpit_vmfd_change_notifier); + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +do_pit_initialize (s, errp); +return 0 ; +} +static void kvm_pit_set_gate (PITCommonState *s,PITChannelState *sc,int val) {kvm_pit_get (s);
@@ -292,6 +309,9 @@ static void kvm_pit_realizefn(DeviceState *dev, Error **errp)qemu_add_vm_change_state_handler (kvm_pit_vm_state_change, s); + s->kvmpit_vmfd_change_notifier.notify = kvmpit_post_vmfd_change; +kvm_vmfd_add_change_notifier (&s->kvmpit_vmfd_change_notifier); + kpc->parent_realize (dev, errp); }
Signature · 1 line
2.42.0
Re: [PATCH v3 24/33] kvm/i8254: add support for confidential guest reset
Quoted 5 lines · Ani Sinha
A confidential guest reset involves closing the old virtual machine KVM file descriptor and opening a new one. Since its a new KVM fd, PIT needs to be reinitialized again. This is done with the help of a notifier which is invoked upon KVM vm file desciptor change during confidential guest reset process.
^^^^^^^^
will fix the above typo and also add a functional test to exercise the changes in this patch for the next spin up.
Quoted 54 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/i386/kvm/i8254.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)hw/i386/kvm/i8254.c +20
-1@@ -52,6 +52,8 @@ struct KVMPITState {LostTickPolicy lost_tick_policy;bool vm_stopped;int64_t kernel_clock_offset; + +NotifierWithReturn kvmpit_vmfd_change_notifier; };struct KVMPITClass {@@ -203,6 +205,21 @@ static void kvm_pit_put(PITCommonState *pit) } } +static int kvmpit_post_vmfd_change (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +KVMPITState *s =container_of (notifier, KVMPITState, + kvmpit_vmfd_change_notifier); + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +do_pit_initialize (s, errp); +return 0 ; +} +static void kvm_pit_set_gate (PITCommonState *s,PITChannelState *sc,int val) {kvm_pit_get (s);@@ -292,6 +309,9 @@ static void kvm_pit_realizefn(DeviceState *dev, Error **errp)qemu_add_vm_change_state_handler (kvm_pit_vm_state_change, s); + s->kvmpit_vmfd_change_notifier.notify = kvmpit_post_vmfd_change; +kvm_vmfd_add_change_notifier (&s->kvmpit_vmfd_change_notifier); + kpc->parent_realize (dev, errp); } --2.42.0
PATCH v3 25/33 hw/hyperv/vmbus: add support for confidential guest reset
In reply to Ani SinhaOn confidential guests when the KVM virtual machine file descriptor changes as a part of the reset process, event file descriptors needs to be reassociated with the new KVM VM file descriptor. This is achieved with the help of a callback handler that gets called when KVM VM file descriptor changes during the confidential guest reset process.
This patch is untested on confidential guests and only exists for completeness.
--- hw/hyperv/vmbus.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
hw/hyperv/vmbus.c +36 -0
@@ -20,6 +20,7 @@#include "hw/hyperv/vmbus-bridge.h" #include "hw/core/sysbus.h" #include "exec/cpu-common.h" +#include "system/kvm.h" #include "exec/target_page.h" #include "trace.h"
@@ -248,6 +249,12 @@ struct VMBus {* interrupt page */ EventNotifier notifier; + +/* +* Notifier to inform when vmfd is changed as a part of confidential guest +* reset mechanism. +*/ +NotifierWithReturn vmbus_vmfd_change_notifier; };static bool gpadl_full (VMBusGpadl *gpadl)
@@ -2347,6 +2354,32 @@ static void vmbus_dev_unrealize(DeviceState *dev)free_channels (vdev); } +/* +* If the KVM fd changes because of VM reset in confidential guests, +* reassociate event fd with the new KVM fd. +*/ +static int vmbus_handle_vmfd_change (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +VMBus *vmbus =container_of (notifier, VMBus, + vmbus_vmfd_change_notifier); +int ret =0 ; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + + ret =hyperv_set_event_flag_handler (VMBUS_EVENT_CONNECTION_ID , + &vmbus->notifier); +/* if we are only using userland event handler, it may already exist */ +if (ret !=0 && ret != -EEXIST ) { +error_setg (errp,"hyperv set event handler failed with %d" , ret); + } + +return ret; +} +static const Property vmbus_dev_props[] = {DEFINE_PROP_UUID ("instanceid" , VMBusDevice, instanceid), };
@@ -2429,6 +2462,9 @@ static void vmbus_realize(BusState *bus, Error **errp) goto clear_event_notifier; } + vmbus->vmbus_vmfd_change_notifier.notify = vmbus_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&vmbus->vmbus_vmfd_change_notifier); +return ; clear_event_notifier:
Signature · 1 line
2.42.0
Re: [PATCH v3 25/33] hw/hyperv/vmbus: add support for confidential guest reset
Quoted 8 lines · Ani Sinha
On confidential guests when the KVM virtual machine file descriptor changes as a part of the reset process, event file descriptors needs to be reassociated with the new KVM VM file descriptor. This is achieved with the help of a callback handler that gets called when KVM VM file descriptor changes during the confidential guest reset process.
This patch is untested on confidential guests and only exists for completeness.
I have been able to exercise this code in a non-coco environment and also will add a functional test for it in the next spin-up. So I will adjust this comment to reflect that the code has been exercised/tested in non-coco and will keep the change.
Quoted 77 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/hyperv/vmbus.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)hw/hyperv/vmbus.c +36
-1@@ -20,6 +20,7 @@#include "hw/hyperv/vmbus-bridge.h" #include "hw/core/sysbus.h" #include "exec/cpu-common.h" +#include "system/kvm.h" #include "exec/target_page.h" #include "trace.h" @@ -248,6 +249,12 @@ struct VMBus {* interrupt page */ EventNotifier notifier; + +/* +* Notifier to inform when vmfd is changed as a part of confidential guest +* reset mechanism. +*/ +NotifierWithReturn vmbus_vmfd_change_notifier; };static bool gpadl_full (VMBusGpadl *gpadl)@@ -2347,6 +2354,32 @@ static void vmbus_dev_unrealize(DeviceState *dev)free_channels (vdev); } +/* +* If the KVM fd changes because of VM reset in confidential guests, +* reassociate event fd with the new KVM fd. +*/ +static int vmbus_handle_vmfd_change (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +VMBus *vmbus =container_of (notifier, VMBus, + vmbus_vmfd_change_notifier); +int ret =0 ; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + + ret =hyperv_set_event_flag_handler (VMBUS_EVENT_CONNECTION_ID , + &vmbus->notifier); +/* if we are only using userland event handler, it may already exist */ +if (ret !=0 && ret != -EEXIST ) { +error_setg (errp,"hyperv set event handler failed with %d" , ret); + } + +return ret; +} +static const Property vmbus_dev_props[] = {DEFINE_PROP_UUID ("instanceid" , VMBusDevice, instanceid), };@@ -2429,6 +2462,9 @@ static void vmbus_realize(BusState *bus, Error **errp) goto clear_event_notifier; } + vmbus->vmbus_vmfd_change_notifier.notify = vmbus_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&vmbus->vmbus_vmfd_change_notifier); +return ; clear_event_notifier: --2.42.0
PATCH v3 26/33 kvm/xen-emu: re-initialize capabilities during confidential guest reset
In reply to Ani SinhaOn confidential guests KVM virtual machine file descriptor changes as a part of the guest reset process. Xen capabilities needs to be re-initialized in KVM against the new file descriptor.
--- target/i386/kvm/xen-emu.c | 50 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)
target/i386/kvm/xen-emu.c +48 -2
@@ -44,9 +44,12 @@#include "xen-compat.h" +NotifierWithReturn xen_vmfd_change_notifier; +static bool hyperv_enabled;static void xen_vcpu_singleshot_timer_event (void *opaque);static void xen_vcpu_periodic_timer_event (void *opaque);static int vcpuop_stop_singleshot_timer (CPUState *cs); +static int do_initialize_xen_caps (KVMState *s,uint32_t hypercall_msr);#ifdef TARGET_X86_64 #define hypercall_compat32 (longmode) (!(longmode))
@@ -54,6 +57,30 @@ static int vcpuop_stop_singleshot_timer(CPUState *cs);#define hypercall_compat32 (longmode) (false)#endif +static int xen_handle_vmfd_change (NotifierWithReturn *n, +void *data,Error ** errp) +{ +int ret; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + + ret =do_initialize_xen_caps (kvm_state,XEN_HYPERCALL_MSR ); +if (ret <0 ) { +return ret; + } + +if (hyperv_enabled) { + ret =do_initialize_xen_caps (kvm_state,XEN_HYPERCALL_MSR_HYPERV ); +if (ret <0 ) { +return ret; + } + } +return 0 ; +} +static bool kvm_gva_to_gpa (CPUState *cs,uint64_t gva,uint64_t *gpa,size_t *len,bool is_write) {
@@ -111,15 +138,16 @@ static inline int kvm_copy_to_gva(CPUState *cs, uint64_t gva, void *buf,return kvm_gva_rw (cs, gva, buf, sz, true); } -int kvm_xen_init (KVMState *s,uint32_t hypercall_msr) +static int do_initialize_xen_caps (KVMState *s,uint32_t hypercall_msr) { +int xen_caps, ret;const int required_caps =KVM_XEN_HVM_CONFIG_HYPERCALL_MSR |KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL |KVM_XEN_HVM_CONFIG_SHARED_INFO ; +struct kvm_xen_hvm_config cfg = { .msr = hypercall_msr, .flags =KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL , }; -int xen_caps, ret; xen_caps =kvm_check_extension (s,KVM_CAP_XEN_HVM );if (required_caps & ~xen_caps) {
@@ -143,6 +171,21 @@ int kvm_xen_init(KVMState *s, uint32_t hypercall_msr) strerror(-ret));return ret; } +return xen_caps; +} + +int kvm_xen_init (KVMState *s,uint32_t hypercall_msr) +{ +int xen_caps; + + xen_caps =do_initialize_xen_caps (s, hypercall_msr); +if (xen_caps <0 ) { +return xen_caps; + } + +if (!hyperv_enabled && (hypercall_msr ==XEN_HYPERCALL_MSR_HYPERV )) { + hyperv_enabled = true; + }/* If called a second time, don't repeat the rest of the setup. */ if (s->xen_caps) {
@@ -185,6 +228,9 @@ int kvm_xen_init(KVMState *s, uint32_t hypercall_msr)xen_primary_console_reset ();xen_xenstore_reset (); + xen_vmfd_change_notifier.notify = xen_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&xen_vmfd_change_notifier); +return 0 ; }
Signature · 1 line
2.42.0
PATCH v3 27/33 kvm/xen_evtchn: add support for confidential guest reset
In reply to Ani SinhaAs a part of the confidential guest reset, when the KVM VM file handle is changed, Xen event ports and kernel ports that were associated with the previous KVM file handle needs to be reassociated with the new handle. This is performed with the help of a callback handler that gets invoked during the confidential guest reset process when the KVM VM file fd changes.
--- hw/i386/kvm/xen_evtchn.c | 113 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 110 insertions(+), 3 deletions(-)
hw/i386/kvm/xen_evtchn.c +110 -3
@@ -133,6 +133,26 @@ struct pirq_info {bool is_translated; }; +struct eventfds { +uint16_t type; +evtchn_port_t port; +int fd; +QLIST_ENTRY (eventfds) node; +}; + +struct kernel_ports { +uint16_t type; +evtchn_port_t port; +uint32_t vcpu_id; +QLIST_ENTRY (kernel_ports) node; +}; + +static QLIST_HEAD(, eventfds) eventfd_list = +QLIST_HEAD_INITIALIZER (eventfd_list); + +static QLIST_HEAD(, kernel_ports) kernel_port_list = +QLIST_HEAD_INITIALIZER (kernel_port_list); +struct XenEvtchnState {/*< private >*/ SysBusDevice busdev;
@@ -178,6 +198,7 @@ struct XenEvtchnState {#define pirq_inuse (s, pirq) (pirq_inuse_word(s, pirq) & pirq_inuse_bit(pirq))struct XenEvtchnState *xen_evtchn_singleton; +static NotifierWithReturn xen_eventchn_notifier;/* Top bits of callback_param are the type (HVM_PARAM_CALLBACK_TYPE_xxx) */ #define CALLBACK_VIA_TYPE_SHIFT 56
@@ -304,6 +325,57 @@ static void gsi_assert_bh(void *opaque) } } +static int xen_eventchn_handle_vmfd_change (NotifierWithReturn *notifier, +void *data,Error **errp) +{ +struct eventfds *ef; +struct kernel_ports *kp; +struct kvm_xen_hvm_attr ha; +CPUState *cpu; +int ret; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + + QLIST_FOREACH(ef, &eventfd_list, node) { + ha.type =KVM_XEN_ATTR_TYPE_EVTCHN ; + ha.u.evtchn.send_port = ef->port; + ha.u.evtchn.type = ef->type; + ha.u.evtchn.flags =0 ; + ha.u.evtchn.deliver.eventfd.port =0 ; + ha.u.evtchn.deliver.eventfd.fd = ef->fd; + + ret =kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); +if (ret <0 ) { +error_setg (errp,"KVM_XEN_HVM_SET_ATTR failed with %d" , ret); +return ret; + } + } + +memset (&ha,0 ,sizeof (ha)); + + QLIST_FOREACH(kp, &kernel_port_list, node) { + cpu =qemu_get_cpu (kp->vcpu_id); + ha.type =KVM_XEN_ATTR_TYPE_EVTCHN ; + ha.u.evtchn.send_port = kp->port; + ha.u.evtchn.type = kp->type; + ha.u.evtchn.flags =0 ; + ha.u.evtchn.deliver.port.port = kp->port; + ha.u.evtchn.deliver.port.vcpu =kvm_arch_vcpu_id (cpu); + ha.u.evtchn.deliver.port.priority = +KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL ; + + ret =kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); +if (ret <0 ) { +error_setg (errp,"KVM_XEN_HVM_SET_ATTR failed with %d" , ret); +return ret; + } + } +return 0 ; +} +void xen_evtchn_create (unsigned int nr_gsis,qemu_irq *system_gsis) { XenEvtchnState *s = XEN_EVTCHN(sysbus_create_simple(TYPE_XEN_EVTCHN,
@@ -350,6 +422,9 @@ void xen_evtchn_create(unsigned int nr_gsis, qemu_irq *system_gsis)/* Set event channel functions for backend drivers to use */ xen_evtchn_ops = &emu_evtchn_backend_ops; + + xen_eventchn_notifier.notify = xen_eventchn_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&xen_eventchn_notifier); }static void xen_evtchn_register_types (void )
@@ -547,6 +622,8 @@ static void inject_callback(XenEvtchnState *s, uint32_t vcpu)static void deassign_kernel_port (evtchn_port_t port) {struct kvm_xen_hvm_attr ha; +struct kernel_ports *kp; +struct eventfds *ef;int ret; ha.type =KVM_XEN_ATTR_TYPE_EVTCHN ;
@@ -557,6 +634,19 @@ static void deassign_kernel_port(evtchn_port_t port)if (ret) {qemu_log_mask (LOG_GUEST_ERROR ,"Failed to unbind kernel port %d: %s\n" , port,strerror (ret)); + }else { + QLIST_FOREACH(kp, &kernel_port_list, node) { +if (kp->port == port) { +QLIST_REMOVE (kp, node); +g_free (kp); + } + } + QLIST_FOREACH(ef, &eventfd_list, node) { +if (ef->port == port) { +QLIST_REMOVE (ef, node); +g_free (ef); + } + } } }
@@ -565,6 +655,8 @@ static int assign_kernel_port(uint16_t type, evtchn_port_t port, {CPUState *cpu =qemu_get_cpu (vcpu_id);struct kvm_xen_hvm_attr ha; +struct kernel_ports *kp =g_malloc0 (sizeof (*kp)); +int ret;if (!cpu) {return -ENOENT ;
@@ -578,12 +670,21 @@ static int assign_kernel_port(uint16_t type, evtchn_port_t port, ha.u.evtchn.deliver.port.vcpu =kvm_arch_vcpu_id (cpu); ha.u.evtchn.deliver.port.priority =KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL ; -return kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); + ret =kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); +if (ret ==0 ) { + kp->type = type; + kp->port = port; + kp->vcpu_id = vcpu_id; +QLIST_INSERT_HEAD (&kernel_port_list, kp, node); + } +return ret; }static int assign_kernel_eventfd (uint16_t type,evtchn_port_t port,int fd) {struct kvm_xen_hvm_attr ha; +struct eventfds *ef =g_malloc0 (sizeof (*ef)); +int ret; ha.type =KVM_XEN_ATTR_TYPE_EVTCHN ; ha.u.evtchn.send_port = port;
@@ -592,7 +693,14 @@ static int assign_kernel_eventfd(uint16_t type, evtchn_port_t port, int fd) ha.u.evtchn.deliver.eventfd.port =0 ; ha.u.evtchn.deliver.eventfd.fd = fd; -return kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); + ret =kvm_vm_ioctl (kvm_state,KVM_XEN_HVM_SET_ATTR , &ha); +if (ret ==0 ) { + ef->type = type; + ef->port = port; + ef->fd = fd; +QLIST_INSERT_HEAD (&eventfd_list, ef, node); + } +return ret; }static bool valid_port (evtchn_port_t port)
@@ -2391,4 +2499,3 @@ void hmp_xen_event_inject(Monitor *mon, const QDict *qdict)monitor_printf (mon,"Delivered port %d\n" , port); } } -
Signature · 1 line
2.42.0
Re: [PATCH v3 27/33] kvm/xen_evtchn: add support for confidential guest reset
Quoted 8 lines · Ani Sinha
+ } else {
+QLIST_FOREACH(kp, &kernel_port_list, node){ + if (kp->port== port) {
+QLIST_REMOVE(kp, node);
+g_free(kp);
+ }
+ }
+QLIST_FOREACH(ef, &eventfd_list, node){
Quoted 8 lines
+ if (
ef->port== port) {
+QLIST_REMOVE(ef, node);
+g_free(ef);
+ }
+ }
}
}
Do those not need to be QLIST_FOREACH_SAFE ?
Quoted 15 lines
hw/i386/kvm/xen_evtchn.c +2
-0@@ -565,6 +655,8 @@ static int assign_kernel_port(uint16_t type, evtchn_port_t port, { CPUState *cpu = qemu_get_cpu(vcpu_id); struct kvm_xen_hvm_attr ha; + struct kernel_ports *kp = g_malloc0(sizeof(*kp)); + int ret;
if (!cpu) {
return-ENOENT;hw/i386/kvm/xen_evtchn.c +0
-0@@ -578,12 +670,21 @@ static int assign_kernel_port(uint16_t type, evtchn_port_t port, ha.u.evtchn.deliver.port.vcpu = kvm_arch_vcpu_id(cpu); ha.u.evtchn.deliver.port.priority = KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEVEL;
- returnkvm_vm_ioctl(kvm_state, KVM_XEN_HVM_SET_ATTR, &ha); + ret =kvm_vm_ioctl(kvm_state, KVM_XEN_HVM_SET_ATTR, &ha);
Quoted 9 lines
+ if (ret == 0) {
+kp->type= type;
+kp->port= port;
+kp->vcpu_id=vcpu_id;
+QLIST_INSERT_HEAD(&kernel_port_list, kp, node); + }
+ return ret;
}
I think 'kp' leaks in the error case there? And the next one in assign_kernel_eventfd().
I don't much like duplicating the code which performs the ioctls. Couldn't you have a helper function which does that when given a 'struct kernel_ports', then invoke *that* from both places?
Or even better, make use of the fact that we *already* have the ability to replay these for serialisation/deserialisation?
But backing up a little more... if this is for rebooting guests, shouldn't we be wiping the event channel setup completely, like a soft reset? Do we preserve to do *any* of this? Are you sure you don't see a soft reset wiping it all away, either before or after this code runs?
Re: [PATCH v3 27/33] kvm/xen_evtchn: add support for confidential guest reset
Quoted 11 lines · David Woodhouse
Quoted 3 lines
I think 'kp' leaks in the error case there? And the next one in
assign_kernel_eventfd().I don't much like duplicating the code which performs the ioctls. Couldn't you have a helper function which does that when given a 'struct
kernel_ports', then invoke *that* from both places?
Quoted 7 lines
Or even better, make use of the fact that we *already* have the ability to replay these for serialisation/deserialisation?
But backing up a little more... if this is for rebooting guests, shouldn't we be wiping the event channel setup completely, like a soft reset? Do we preserve to do *any* of this? Are you sure you don't see a soft reset wiping it all away, either before or after this code runs?
Ah yes, good catch! I added the following assertions and they never failed has I walked through the code with gdb:
hw/i386/kvm/xen_evtchn.c +3 -0
@@ -1241,6 +1241,9 @@ int xen_evtchn_soft_reset(void)kvm_update_msi_routes_all (NULL , true,0 ,0 ); } +assert (QLIST_EMPTY (&eventfd_list)); +assert (QLIST_EMPTY (&kernel_port_list)); +return 0 ; }
I reverted this patch and it does not break my Xen functional test. So it seems the soft reset callsxen_evtchn_soft_reset() -> close_port() -> deassign_kernel_port() and all ports are cleaned up, meaning they do not need preserving through the reset.
I will drop this patch in the next spin-up.
PATCH v3 28/33 ppc/openpic: create a new openpic device and reattach mem region on coco reset
In reply to Ani SinhaFor confidential guests during the reset process, the old KVM VM file descriptor is closed and a new one is created. When a new file descriptor is created, a new openpic device needs to be created against this new KVM VM file descriptor as well. Additionally, existing memory region needs to be reattached to this new openpic device and proper CPU attributes set associating new file descriptor. This change makes this happen with the help of a callback handler that gets called when the KVM VM file descriptor changes as a part of the confidential guest reset process.
--- hw/intc/openpic_kvm.c | 112 +++++++++++++++++++++++++++++++++--------- 1 file changed, 88 insertions(+), 24 deletions(-)
hw/intc/openpic_kvm.c +88 -24
@@ -49,6 +49,7 @@ struct KVMOpenPICState {uint32_t fd;uint32_t model;hwaddr mapped; +NotifierWithReturn vmfd_change_notifier; };static void kvm_openpic_set_irq (void *opaque,int n_IRQ,int level)
@@ -114,6 +115,88 @@ static const MemoryRegionOps kvm_openpic_mem_ops = { }, }; +static int kvm_openpic_setup (KVMOpenPICState *opp,Error **errp) +{ +int kvm_openpic_model; +struct kvm_create_device cd = {0 }; +KVMState *s = kvm_state; +int ret; + +switch (opp->model) { +case OPENPIC_MODEL_FSL_MPIC_20 : + kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_20 ; +break ; + +case OPENPIC_MODEL_FSL_MPIC_42 : + kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_42 ; +break ; + +default : +error_setg (errp,"Unsupported OpenPIC model %" PRIu32, opp->model); +return -1 ; + } + + cd.type = kvm_openpic_model; + ret =kvm_vm_ioctl (s,KVM_CREATE_DEVICE , &cd); +if (ret <0 ) { +error_setg (errp,"Can't create device %d: %s" , + cd.type,strerror (errno)); +return -1 ; + } + opp->fd = cd.fd; + +return 0 ; +} + +static int kvm_openpic_handle_vmfd_change (NotifierWithReturn *notifier, +void *data,Error **errp) +{ +KVMOpenPICState *opp =container_of (notifier, KVMOpenPICState, + vmfd_change_notifier); +uint64_t reg_base; +struct kvm_device_attr attr; +CPUState *cs; +int ret; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +/* close the old descriptor */ +close (opp->fd); + +if (kvm_openpic_setup (opp, errp) <0 ) { +return -1 ; + } + +if (!opp->mapped) { +return 0 ; + } + + reg_base = opp->mapped; + attr.group =KVM_DEV_MPIC_GRP_MISC ; + attr.attr =KVM_DEV_MPIC_BASE_ADDR ; + attr.addr = (uint64_t )(unsigned long )®_base; + + ret =ioctl (opp->fd,KVM_SET_DEVICE_ATTR , &attr); +if (ret <0 ) { +error_setg (errp,"%s: %s %" PRIx64, __func__, +strerror (errno), reg_base); +return -1 ; + } + +CPU_FOREACH (cs) { + ret =kvm_vcpu_enable_cap (cs,KVM_CAP_IRQ_MPIC ,0 , opp->fd, +kvm_arch_vcpu_id (cs)); +if (ret <0 ) { +return ret; + } + } + +return 0 ; +} +static void kvm_openpic_region_add (MemoryListener *listener,MemoryRegionSection *section) {
@@ -197,36 +280,14 @@ static void kvm_openpic_realize(DeviceState *dev, Error **errp)SysBusDevice *d =SYS_BUS_DEVICE (dev);KVMOpenPICState *opp =KVM_OPENPIC (dev);KVMState *s = kvm_state; -int kvm_openpic_model; -struct kvm_create_device cd = {0 }; -int ret, i; +int i;if (!kvm_check_extension (s,KVM_CAP_DEVICE_CTRL )) {error_setg (errp,"Kernel is lacking Device Control API" );return ; } -switch (opp->model) { -case OPENPIC_MODEL_FSL_MPIC_20 : - kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_20 ; -break ; - -case OPENPIC_MODEL_FSL_MPIC_42 : - kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_42 ; -break ; - -default : -error_setg (errp,"Unsupported OpenPIC model %" PRIu32, opp->model); -return ; - } - - cd.type = kvm_openpic_model; - ret =kvm_vm_ioctl (s,KVM_CREATE_DEVICE , &cd); -if (ret <0 ) { -error_setg_errno (errp, errno,"Can't create device %d" , cd.type); -return ; - } - opp->fd = cd.fd; +kvm_openpic_setup (opp, errp);sysbus_init_mmio (d, &opp->mem);qdev_init_gpio_in (dev, kvm_openpic_set_irq,OPENPIC_MAX_IRQ );
@@ -235,6 +296,9 @@ static void kvm_openpic_realize(DeviceState *dev, Error **errp) opp->mem_listener.region_del = kvm_openpic_region_del; opp->mem_listener.name ="openpic-kvm" ;memory_listener_register (&opp->mem_listener, &address_space_memory); + opp->vmfd_change_notifier.notify = + kvm_openpic_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&opp->vmfd_change_notifier);/* indicate pic capabilities */ msi_nonbroken = true;
Signature · 1 line
2.42.0
Re: [PATCH v3 28/33] ppc/openpic: create a new openpic device and reattach mem region on coco reset
Am 27. Januar 2026 05:15:56 UTC schrieb Ani Sinha <[email protected]>:
Quoted 155 lines
For confidential guests during the reset process, the old KVM VM file descriptor is closed and a new one is created. When a new file descriptor is created, a new openpic device needs to be created against this new KVM VM file descriptor as well. Additionally, existing memory region needs to be reattached to this new openpic device and proper CPU attributes set associating new file descriptor. This change makes this happen with the help of a callback handler that gets called when the KVM VM file descriptor changes as a part of the confidential guest reset process.
Signed-off-by Ani Sinha <[email protected]>---
hw/intc/openpic_kvm.c| 112 +++++++++++++++++++++++++++++++++---------1 file changed, 88 insertions(+), 24 deletions(-)hw/intc/openpic_kvm.c +85
-24@@ -49,6 +49,7 @@ struct KVMOpenPICState {uint32_t fd;uint32_t model;hwaddr mapped; +NotifierWithReturn vmfd_change_notifier; };static void kvm_openpic_set_irq (void *opaque,int n_IRQ,int level)@@ -114,6 +115,88 @@ static const MemoryRegionOps kvm_openpic_mem_ops = { }, }; +static int kvm_openpic_setup (KVMOpenPICState *opp,Error **errp) +{ +int kvm_openpic_model; +struct kvm_create_device cd = {0 }; +KVMState *s = kvm_state; +int ret; + +switch (opp->model) { +case OPENPIC_MODEL_FSL_MPIC_20 : + kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_20 ; +break ; + +case OPENPIC_MODEL_FSL_MPIC_42 : + kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_42 ; +break ; + +default : +error_setg (errp,"Unsupported OpenPIC model %" PRIu32, opp->model); +return -1 ; + } + + cd.type = kvm_openpic_model; + ret =kvm_vm_ioctl (s,KVM_CREATE_DEVICE , &cd); +if (ret <0 ) { +error_setg (errp,"Can't create device %d: %s" , + cd.type,strerror (errno)); +return -1 ; + } + opp->fd = cd.fd; + +return 0 ; +} + +static int kvm_openpic_handle_vmfd_change (NotifierWithReturn *notifier, +void *data,Error **errp) +{ +KVMOpenPICState *opp =container_of (notifier, KVMOpenPICState, + vmfd_change_notifier); +uint64_t reg_base; +struct kvm_device_attr attr; +CPUState *cs; +int ret; + +/* we are not interested in pre vmfd change notification */ +if (((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } + +/* close the old descriptor */ +close (opp->fd); + +if (kvm_openpic_setup (opp, errp) <0 ) { +return -1 ; + } + +if (!opp->mapped) { +return 0 ; + } + + reg_base = opp->mapped; + attr.group =KVM_DEV_MPIC_GRP_MISC ; + attr.attr =KVM_DEV_MPIC_BASE_ADDR ; + attr.addr = (uint64_t )(unsigned long )®_base; + + ret =ioctl (opp->fd,KVM_SET_DEVICE_ATTR , &attr); +if (ret <0 ) { +error_setg (errp,"%s: %s %" PRIx64, __func__, +strerror (errno), reg_base); +return -1 ; + } + +CPU_FOREACH (cs) { + ret =kvm_vcpu_enable_cap (cs,KVM_CAP_IRQ_MPIC ,0 , opp->fd, +kvm_arch_vcpu_id (cs)); +if (ret <0 ) { +return ret; + } + } + +return 0 ; +} +static void kvm_openpic_region_add (MemoryListener *listener,MemoryRegionSection *section) {@@ -197,36 +280,14 @@ static void kvm_openpic_realize(DeviceState *dev, Error **errp)SysBusDevice *d =SYS_BUS_DEVICE (dev);KVMOpenPICState *opp =KVM_OPENPIC (dev);KVMState *s = kvm_state; -int kvm_openpic_model; -struct kvm_create_device cd = {0 }; -int ret, i; +int i;if (!kvm_check_extension (s,KVM_CAP_DEVICE_CTRL )) {error_setg (errp,"Kernel is lacking Device Control API" );return ; } -switch (opp->model) { -case OPENPIC_MODEL_FSL_MPIC_20 : - kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_20 ; -break ; - -case OPENPIC_MODEL_FSL_MPIC_42 : - kvm_openpic_model =KVM_DEV_TYPE_FSL_MPIC_42 ; -break ; - -default : -error_setg (errp,"Unsupported OpenPIC model %" PRIu32, opp->model); -return ; - } - - cd.type = kvm_openpic_model; - ret =kvm_vm_ioctl (s,KVM_CREATE_DEVICE , &cd); -if (ret <0 ) { -error_setg_errno (errp, errno,"Can't create device %d" , cd.type); -return ; - } - opp->fd = cd.fd; +kvm_openpic_setup (opp, errp);sysbus_init_mmio (d, &opp->mem);qdev_init_gpio_in (dev, kvm_openpic_set_irq,OPENPIC_MAX_IRQ );
hw/intc/openpic_kvm.c static void kvm_openpic_realize(DeviceState *dev, Error **errp) · line 304
Quoted 9 lines
hw/intc/openpic_kvm.c +3
-0@@ -235,6 +296,9 @@ static void kvm_openpic_realize(DeviceState *dev, Error **errp) opp->mem_listener.region_del = kvm_openpic_region_del; opp->mem_listener.name ="openpic-kvm" ;memory_listener_register (&opp->mem_listener, &address_space_memory); + opp->vmfd_change_notifier.notify = + kvm_openpic_handle_vmfd_change; +kvm_vmfd_add_change_notifier (&opp->vmfd_change_notifier);/* indicate pic capabilities */ msi_nonbroken = true;
Code seems structurally comprehensible to me, so:
PATCH v3 29/33 kvm/vcpu: add notifiers to inform vcpu file descriptor change
In reply to Ani SinhaWhen new vcpu file descriptors are created and bound to the new kvm file descriptor as a part of the confidential guest reset mechanism, various subsystems needs to know about it. This change adds notifiers so that various subsystems can take appropriate actions when vcpu fds change by registering their handlers to this notifier.
Subsequent changes will register specific handlers to this notifier.
--- accel/kvm/kvm-all.c | 26 ++++++++++++++++++++++++++ accel/stubs/kvm-stub.c | 10 ++++++++++ include/system/kvm.h | 17 +++++++++++++++++ 3 files changed, 53 insertions(+)
accel/kvm/kvm-all.c +26 -0
@@ -127,6 +127,9 @@ static NotifierList kvm_irqchip_change_notifiers =static NotifierWithReturnList register_vmfd_changed_notifiers =NOTIFIER_WITH_RETURN_LIST_INITIALIZER (register_vmfd_changed_notifiers); +static NotifierWithReturnList register_vcpufd_changed_notifiers = +NOTIFIER_WITH_RETURN_LIST_INITIALIZER (register_vcpufd_changed_notifiers); +static int map_kvm_run (KVMState *s,CPUState *cpu,Error **errp);static int map_kvm_dirty_gfns (KVMState *s,CPUState *cpu,Error **errp);static int vcpu_unmap_regions (KVMState *s,CPUState *cpu);
@@ -2313,6 +2316,22 @@ static int kvm_vmfd_change_notify(Error **errp) &vmfd_notifier, errp); } +void kvm_vcpufd_add_change_notifier (NotifierWithReturn *n) +{ +notifier_with_return_list_add (®ister_vcpufd_changed_notifiers, n); +} + +void kvm_vcpufd_remove_change_notifier (NotifierWithReturn *n) +{ +notifier_with_return_remove (n); +} + +static int kvm_vcpufd_change_notify (Error **errp) +{ +return notifier_with_return_list_notify (®ister_vcpufd_changed_notifiers, + &vmfd_notifier, errp); +} +int kvm_irqchip_get_virq (KVMState *s) {int next_virq;
@@ -2840,6 +2859,13 @@ static int kvm_reset_vmfd(MachineState *ms) }assert (!err); +/* notify everyone that vcpu fd has changed. */ + ret =kvm_vcpufd_change_notify (&err); +if (ret <0 ) { +return ret; + } +assert (!err); +/* these can be only called after ram_block_rebind() */ memory_listener_register (&kml->listener, &address_space_memory);memory_listener_register (&kvm_io_listener, &address_space_io);
accel/stubs/kvm-stub.c +10 -0
@@ -87,6 +87,16 @@ void kvm_vmfd_remove_change_notifier(NotifierWithReturn *n) { } +void kvm_vcpufd_add_change_notifier (NotifierWithReturn *n) +{ +return ; +} + +void kvm_vcpufd_remove_change_notifier (NotifierWithReturn *n) +{ +return ; +} +int kvm_irqchip_add_irqfd_notifier_gsi (KVMState *s,EventNotifier *n,EventNotifier *rn,int virq) {
include/system/kvm.h +17 -0
@@ -589,4 +589,21 @@ void kvm_vmfd_add_change_notifier(NotifierWithReturn *n);*/ void kvm_vmfd_remove_change_notifier (NotifierWithReturn *n); +/** +* kvm_vcpufd_add_change_notifier - register a notifier to get notified when +* a KVM vcpu file descriptors changes as a part of the confidential guest +* "reset" process. Various subsystems should use this mechanism to take +* actions such as re-issuing vcpu ioctls as a part of setting up vcpu +* features. +* @n: notifier with return value. +*/ +void kvm_vcpufd_add_change_notifier (NotifierWithReturn *n); + +/** +* kvm_vcpufd_remove_change_notifier - de-register a notifer previously +* registered with kvm_vcpufd_add_change_notifier call. +* @n: notifier that was previously registered. +*/ +void kvm_vcpufd_remove_change_notifier (NotifierWithReturn *n); +#endif
Signature · 1 line
2.42.0
PATCH v3 30/33 kvm/i386/apic: set local apic after vcpu file descriptors changed
In reply to Ani SinhaOnce the vcpu file descriptors changed after confidential guest reset, the local apic needs to be reinitialized. This change adds a callback from the vcpu fd change notifiers to reinitialize local apic for kvm x86.
--- hw/i386/kvm/apic.c | 13 +++++++++++++ include/hw/i386/apic_internal.h | 1 + 2 files changed, 14 insertions(+)
hw/i386/kvm/apic.c +13 -0
@@ -229,6 +229,16 @@ static void kvm_apic_reset(APICCommonState *s)run_on_cpu (CPU (s->cpu), kvm_apic_put,RUN_ON_CPU_HOST_PTR (s)); } +static int apic_vcpufd_change_handler (NotifierWithReturn *n, +void *data,Error ** errp) { +APICCommonState *s =container_of (n, APICCommonState, + vcpufd_change_notifier); + +run_on_cpu (CPU (s->cpu), kvm_apic_put,RUN_ON_CPU_HOST_PTR (s)); + +return 0 ; +} +static void kvm_apic_realize (DeviceState *dev,Error **errp) {APICCommonState *s =APIC_COMMON (dev);
@@ -238,6 +248,9 @@ static void kvm_apic_realize(DeviceState *dev, Error **errp)assert (kvm_has_gsi_routing ()); msi_nonbroken = true; + + s->vcpufd_change_notifier.notify = apic_vcpufd_change_handler; +kvm_vcpufd_add_change_notifier (&s->vcpufd_change_notifier); }static void kvm_apic_unrealize (DeviceState *dev)
include/hw/i386/apic_internal.h +1 -0
@@ -189,6 +189,7 @@ struct APICCommonState {hwaddr vapic_paddr;/* note: persistence via kvmvapic */ bool legacy_instance_id;uint32_t extended_log_dest; +NotifierWithReturn vcpufd_change_notifier; }; typedef struct VAPICState {
Signature · 1 line
2.42.0
Re: [PATCH v3 30/33] kvm/i386/apic: set local apic after vcpu file descriptors changed
Quoted 4 lines · Ani Sinha
Once the vcpu file descriptors changed after confidential guest reset, the local apic needs to be reinitialized. This change adds a callback from the vcpu fd change notifiers to reinitialize local apic for kvm x86.
This change is not necessary as kvm_apic_reset() does the same thing as a part of the reset process. Will drop this patch in next spin up.
Quoted 53 lines
Signed-off-by Ani Sinha <[email protected]>--- hw/i386/kvm/apic.c | 13 +++++++++++++ include/hw/i386/apic_internal.h | 1 + 2 files changed, 14 insertions(+)hw/i386/kvm/apic.c +13
-0@@ -229,6 +229,16 @@ static void kvm_apic_reset(APICCommonState *s)run_on_cpu (CPU (s->cpu), kvm_apic_put,RUN_ON_CPU_HOST_PTR (s)); } +static int apic_vcpufd_change_handler (NotifierWithReturn *n, +void *data,Error ** errp) { +APICCommonState *s =container_of (n, APICCommonState, + vcpufd_change_notifier); + +run_on_cpu (CPU (s->cpu), kvm_apic_put,RUN_ON_CPU_HOST_PTR (s)); + +return 0 ; +} +static void kvm_apic_realize (DeviceState *dev,Error **errp) {APICCommonState *s =APIC_COMMON (dev);@@ -238,6 +248,9 @@ static void kvm_apic_realize(DeviceState *dev, Error **errp)assert (kvm_has_gsi_routing ()); msi_nonbroken = true; + + s->vcpufd_change_notifier.notify = apic_vcpufd_change_handler; +kvm_vcpufd_add_change_notifier (&s->vcpufd_change_notifier); }static void kvm_apic_unrealize (DeviceState *dev)include/hw/i386/apic_internal.h +1
-1@@ -189,6 +189,7 @@ struct APICCommonState {hwaddr vapic_paddr;/* note: persistence via kvmvapic */ bool legacy_instance_id;uint32_t extended_log_dest; +NotifierWithReturn vcpufd_change_notifier; }; typedef struct VAPICState { --2.42.0
PATCH v3 31/33 kvm/clock: add support for confidential guest reset
In reply to Ani SinhaConfidential guests change the KVM VM file descriptor upon reset and also create new VCPU file descriptors against the new KVM VM file descriptor. We need to save the clock state from kvm before KVM VM file descriptor changes and restore it after. Also after VCPU file descriptors changed, we must call KVM_KVMCLOCK_CTRL on the VCPU file descriptor to inform KVM that the VCPU is in paused state.
--- hw/i386/kvm/clock.c | 59 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+)
hw/i386/kvm/clock.c +59 -0
@@ -50,6 +50,9 @@ struct KVMClockState {/* whether the 'clock' value was obtained in a host with * reliable KVM_GET_CLOCK */ bool clock_is_reliable; + +NotifierWithReturn kvmclock_vcpufd_change_notifier; +NotifierWithReturn kvmclock_vmfd_change_notifier; };struct pvclock_vcpu_time_info {
@@ -63,6 +66,9 @@ struct pvclock_vcpu_time_info {uint8_t pad[2 ]; } __attribute__((__packed__)); /* 32 bytes */ +static int kvmclock_set_clock (NotifierWithReturn *notifier, +void *data,Error ** errp); +static uint64_t kvmclock_current_nsec (KVMClockState *s) {CPUState *cpu = first_cpu;
@@ -219,6 +225,54 @@ static void kvmclock_vm_state_change(void *opaque, bool running, } } +static int kvmclock_save_clock (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +if (!((VmfdChangeNotifier *)data)->pre) { +return 0 ; + } +KVMClockState *s =container_of (notifier, KVMClockState, + kvmclock_vmfd_change_notifier); +kvm_update_clock (s); +return 0 ; +} + +static int kvmclock_set_clock (NotifierWithReturn *notifier, +void *data,Error ** errp) +{ +struct kvm_clock_data clock_data = {}; +CPUState *cpu; +int ret; +KVMClockState *s =container_of (notifier, KVMClockState, + kvmclock_vcpufd_change_notifier); +int cap_clock_ctrl =kvm_check_extension (kvm_state,KVM_CAP_KVMCLOCK_CTRL ); + +if (!s->clock_is_reliable) { +uint64_t pvclock_via_mem =kvmclock_current_nsec (s); +/* saved clock value before vmfd change is not reliable */ +if (pvclock_via_mem) { + s->clock = pvclock_via_mem; + } + } + + clock_data.clock = s->clock; + ret =kvm_vm_ioctl (kvm_state,KVM_SET_CLOCK , &clock_data); +if (ret <0 ) { +fprintf (stderr,"KVM_SET_CLOCK failed: %s\n" ,strerror (-ret)); +abort (); + } + +if (!cap_clock_ctrl) { +return 0 ; + } +CPU_FOREACH (cpu) { +run_on_cpu (cpu, do_kvmclock_ctrl,RUN_ON_CPU_NULL ); + } + +return 0 ; +} + +static void kvmclock_realize (DeviceState *dev,Error **errp) {KVMClockState *s =KVM_CLOCK (dev);
@@ -230,7 +284,12 @@ static void kvmclock_realize(DeviceState *dev, Error **errp)kvm_update_clock (s); + s->kvmclock_vcpufd_change_notifier.notify = kvmclock_set_clock; + s->kvmclock_vmfd_change_notifier.notify = kvmclock_save_clock; +qemu_add_vm_change_state_handler (kvmclock_vm_state_change, s); +kvm_vcpufd_add_change_notifier (&s->kvmclock_vcpufd_change_notifier); +kvm_vmfd_add_change_notifier (&s->kvmclock_vmfd_change_notifier); }static bool kvmclock_clock_is_reliable_needed (void *opaque)
Signature · 1 line
2.42.0
PATCH v3 32/33 hw/machine: introduce machine specific option 'x-change-vmfd-on-reset'
In reply to Ani SinhaA new machine specific option 'x-change-vmfd-on-reset' is introduced for debugging and testing only (hence the 'x-' prefix). This option when enabled will force KVM VM file descriptor to be changed upon guest reset like in the case of confidential guests. This can be used to exercize the code changes that are specific for confidential guests on non-confidential guests as well (except changes that require hardware support for confidential guests).
A new functional test has been added in the next patch that uses this new parameter to test the VM file descriptor changes.
--- hw/core/machine.c | 22 ++++++++++++++++++++++ include/hw/core/boards.h | 6 ++++++ system/runstate.c | 17 +++++++++-------- 3 files changed, 37 insertions(+), 8 deletions(-)
hw/core/machine.c +22 -0
@@ -450,6 +450,21 @@ static void machine_set_dump_guest_core(Object *obj, bool value, Error **errp) ms->dump_guest_core = value; } +static bool machine_get_new_accel_vmfd_on_reset (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->new_accel_vmfd_on_reset; +} + +static void machine_set_new_accel_vmfd_on_reset (Object *obj, +bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->new_accel_vmfd_on_reset = value; +} +static bool machine_get_mem_merge (Object *obj,Error **errp) {MachineState *ms =MACHINE (obj);
@@ -1198,6 +1213,13 @@ static void machine_class_init(ObjectClass *oc, const void *data)object_class_property_set_description (oc,"dump-guest-core" ,"Include guest memory in a core dump" ); +object_class_property_add_bool (oc,"x-change-vmfd-on-reset" , + machine_get_new_accel_vmfd_on_reset, + machine_set_new_accel_vmfd_on_reset); +object_class_property_set_description (oc,"x-change-vmfd-on-reset" , +"Generate new accelerator fd on reset, " +"to be used only for testing and debugging." ); +object_class_property_add_bool (oc,"mem-merge" , machine_get_mem_merge, machine_set_mem_merge); object_class_property_set_description(oc, "mem-merge",
include/hw/core/boards.h +6 -0
@@ -447,6 +447,12 @@ struct MachineState {struct NVDIMMState *nvdimms_state;struct NumaState *numa_state;bool acpi_spcr_enabled; +/* +* whether to change virtual machine accelerator file descriptor upon +* reset or not. used only for debugging and testing purpose. +* It should be set to false for all regular use. +*/ +bool new_accel_vmfd_on_reset; };/*
system/runstate.c +9 -8
@@ -525,22 +525,23 @@ void qemu_system_reset(ShutdownCause reason) type =RESET_TYPE_COLD ; } -/* -* different accelerators implement how to close the old file handle of -* the accelerator descriptor and create a new one here. Resetting -* file handle is necessary to create a new confidential VM context post -* VM reset. -*/ -if (!cpus_are_resettable () && reason ==SHUTDOWN_CAUSE_GUEST_RESET ) { +if (reason ==SHUTDOWN_CAUSE_GUEST_RESET && + (current_machine->new_accel_vmfd_on_reset || !cpus_are_resettable ())) {if (ac->reset_vmfd) { ret = ac->reset_vmfd (current_machine);if (ret <0 ) {error_report ("unable to reset vmfd: %s(%d)" ,strerror (-ret), ret);vm_stop (RUN_STATE_INTERNAL_ERROR ); + }else if (current_machine->new_accel_vmfd_on_reset) { +info_report ("virtual machine accel file descriptor " +"has changed." ); } + }else if (!cpus_are_resettable ()) { +error_report ("accelerator does not support reset!" ); }else { -error_report ("accelerator does not support reset" ); +error_report ("accelerator does not support vmfd change reset, " +"proceeding with normal reset!" ); } }
Signature · 1 line
2.42.0
PATCH v3 33/33 tests/functional/x86_64: add functional test to exercise vm fd change on reset
In reply to Ani SinhaA new functional test is added that exercises the code changes related to closing of the old KVM VM file descriptor and opening a new one upon VM reset. This normally happens when confidential guests are resetted but for non-confidential guests, we use a special machine specific debug/test parameter 'x-change-vmfd-on-reset' to enable this behavior.
Only specific code changes related to re-initialization of SEV-ES, SEV-SNP and TDX platforms are not exercized in this test as they require hardware that supports running confidential guests.
--- MAINTAINERS | 6 ++ tests/functional/x86_64/meson.build | 1 + .../x86_64/test_vmfd_change_reboot.py | 96 +++++++++++++++++++ 3 files changed, 103 insertions(+) create mode 100755 tests/functional/x86_64/test_vmfd_change_reboot.py
MAINTAINERS +6 -0
@@ -151,6 +151,12 @@ F: tools/i386/ F: tests/functional/i386/ F: tests/functional/x86_64/ +X86 VM file descriptor change on reset test +M: Ani Sinha <[email protected]> +M: Paolo Bonzini <[email protected]> +S: Maintained +F: tests/functional/x86_64/test_vmfd_change_reboot.py + Guest CPU cores (TCG) --------------------- Overall TCG CPUs
tests/functional/x86_64/meson.build +1 -0
@@ -36,4 +36,5 @@ tests_x86_64_system_thorough = [ 'vfio_user_client', 'virtio_balloon', 'virtio_gpu', + 'vmfd_change_reboot', ]
tests/functional/x86_64/test_vmfd_change_reboot.py new file +96 -0
96 lines not on this page · raw message
Signature · 1 line
2.42.0