Add migration support to the MSHV accelerator
Subsystems (2)
cover Add migration support to the MSHV accelerator
Hey all, This is a patch series for live migration support in the MSHV accelerator. Since this is somewhat invasive and touches various parts outside of the accel's folder hierarchies I'm sending an RFC series to collect early feedback.
This patches are sent in sent in isolation, but for LM to be fully functional we will require require the recently submitted patch series "Support QEMU cpu models in MSHV accelerator" (at v6 currently) to be merged, since on some hosts we will need the CPUID infra to disable features that we currently are not able to migrate yet (e.g. AMX tiles).
In this series we perform some preperatory refactorings and introduce new abstractions where required, particulary for irqchips and MSR logic. We also have to introduce some generic logic for XSAVE de/compaction to allow migration of XSAVE state.
Note: there is some pending optimization to avoid a buffer copy and rework existing xsave helper code to be generic over a compacted or standard layout. We'll address this in a later revision or in a discrete patch.
The guest state components that are covered by migration are:
- standard regs
- special regs
- xcr0
- (legacy) FPU regs
- XSAVE
- LAPIC
- MSRs
- SynIC state (SIMP, SIEFP, STIMER)
- pending interrupts/exceptions
- MP state (AP cpu modes)
Finally, routines for dirty-page tracking to reduce migration downtime have beend added and integrated in the respective hooks.
best,
magnus
Changes since RFC:
- Added CET SS/IBT MSR migration
- Assert 64bit padding on CPUX86State->sysenter_cs statically
Magnus Kulke (34):
target/i386/mshv: use arch_load/store_reg fns target/i386/mshv: use generic FPU/xcr0 state target/i386/mshv: impl init/load/store_vcpu_state accel/accel-irq: add AccelRouteChange abstraction accel/accel-irq: add generic begin_route_changes accel/accel-irq: add generic commit_route_changes accel/mshv: add irq_routes to state accel/mshv: update s->irq_routes in add_msi_route accel/mshv: update s->irq_routes in update_msi_route accel/mshv: update s->irq_routes in release_virq accel/mshv: use s->irq_routes in commit_routes accel/mshv: reserve ioapic routes on s->irq_routes accel/mshv: remove redundant msi controller target/i386/mshv: move apic logic into own file target/i386/mshv: remove redundant apic helpers target/i386/mshv: migrate LAPIC state target/i386/mshv: move msr code to arch accel/mshv: store partition proc features target/i386/mshv: expose msvh_get_generic_regs target/i386/mshv: migrate MSRs target/i386/mshv: migrate MTRR MSRs target/i386/mshv: migrate Synic SINT MSRs target/i386/mshv: migrate CET/SS MSRs target/i386/mshv: migrate SIMP and SIEFP state target/i386/mshv: migrate STIMER state accel/mshv: introduce SaveVMHandler accel/mshv: write synthetic MSRs after migration accel/mshv: migrate REFERENCE_TIME target/i386/mshv: migrate pending ints/excs target/i386: add de/compaction to xsave_helper target/i386/mshv: migrate XSAVE state target/i386/mshv: reconstruct hflags after load target/i386/mshv: migrate MP_STATE accel/mshv: enable dirty page tracking
accel/accel-irq.c | 41 +- accel/kvm/kvm-all.c | 6 +- accel/mshv/irq.c | 360 ++++++------ accel/mshv/mem.c | 211 +++++++ accel/mshv/meson.build | 1 - accel/mshv/mshv-all.c | 243 +++++++- accel/mshv/msr.c | 375 ------------- accel/stubs/kvm-stub.c | 2 +- accel/stubs/mshv-stub.c | 6 +- hw/intc/apic_common.c | 3 + hw/misc/ivshmem-pci.c | 8 +- hw/vfio/pci.c | 11 +- hw/virtio/virtio-pci.c | 3 +- include/accel/accel-route.h | 17 + include/hw/hyperv/hvgdk_mini.h | 33 ++ include/hw/hyperv/hvhdk.h | 150 +++++ include/hw/i386/apic_internal.h | 5 + include/system/accel-irq.h | 6 +- include/system/kvm.h | 23 +- include/system/mshv.h | 15 +- include/system/mshv_int.h | 89 +-- target/i386/cpu.h | 14 +- target/i386/kvm/kvm.c | 5 +- target/i386/machine.c | 46 ++ target/i386/mshv/meson.build | 3 + target/i386/mshv/mshv-apic.c | 78 +++ target/i386/mshv/mshv-cpu.c | 958 +++++++++++++++++++++++--------- target/i386/mshv/msr.c | 467 ++++++++++++++++ target/i386/mshv/synic.c | 206 +++++++ target/i386/xsave_helper.c | 255 +++++++++ 30 files changed, 2684 insertions(+), 956 deletions(-) delete mode 100644 accel/mshv/msr.c create mode 100644 include/accel/accel-route.h create mode 100644 target/i386/mshv/mshv-apic.c create mode 100644 target/i386/mshv/msr.c create mode 100644 target/i386/mshv/synic.c
Signature · 1 line
2.34.1
PATCH 1/34 target/i386/mshv: use arch_load/store_reg fns
Improved consistency around the naming of load/store register fn's. this is required since we want to roundtrip more registers in a migration than what's currently required for MMIO emulation.
--- accel/mshv/mshv-all.c | 2 +- include/system/mshv_int.h | 6 ++--- target/i386/mshv/mshv-cpu.c | 52 ++++++++++++++----------------------- 3 files changed, 23 insertions(+), 37 deletions(-)
accel/mshv/mshv-all.c +1 -1
@@ -650,7 +650,7 @@ static void mshv_cpu_synchronize_pre_loadvm(CPUState *cpu)static void do_mshv_cpu_synchronize (CPUState *cpu,run_on_cpu_data arg) {if (!cpu->accel->dirty) { -int ret =mshv_load_regs (cpu); +int ret =mshv_arch_load_regs (cpu);if (ret <0 ) {error_report ("Failed to load registers for vcpu %d" , cpu->cpu_index);
include/system/mshv_int.h +2 -4
@@ -82,11 +82,9 @@ void mshv_init_mmio_emu(void);int mshv_create_vcpu (int vm_fd,uint8_t vp_index,int *cpu_fd);void mshv_remove_vcpu (int vm_fd,int cpu_fd);int mshv_configure_vcpu (const CPUState *cpu,const MshvFPU *fpu,uint64_t xcr0); -int mshv_get_standard_regs (CPUState *cpu); -int mshv_get_special_regs (CPUState *cpu);int mshv_run_vcpu (int vm_fd,CPUState *cpu,hv_message *msg,MshvVmExit *exit); -int mshv_load_regs (CPUState *cpu); -int mshv_store_regs (CPUState *cpu); +int mshv_arch_load_regs (CPUState *cpu); +int mshv_arch_store_regs (CPUState *cpu);int mshv_set_generic_regs (const CPUState *cpu,const hv_register_assoc *assocs,size_t n_regs);int mshv_arch_put_registers (const CPUState *cpu);
target/i386/mshv/mshv-cpu.c +20 -32
128 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 2/34 target/i386/mshv: use generic FPU/xcr0 state
In reply to Magnus KulkeInstead of using an mshv-specific FPU state representation we switch to the generic i386 representation of the registers.
--- include/system/mshv_int.h | 15 +------- target/i386/mshv/mshv-cpu.c | 76 ++++++++++++++++++++++--------------- 2 files changed, 47 insertions(+), 44 deletions(-)
include/system/mshv_int.h +1 -14
@@ -58,19 +58,6 @@ typedef struct MshvMsiControl {#define mshv_vcpufd (cpu) (cpu->accel->cpufd)/* cpu */ -typedef struct MshvFPU { -uint8_t fpr[8 ][16 ]; -uint16_t fcw; -uint16_t fsw; -uint8_t ftwx; -uint8_t pad1; -uint16_t last_opcode; -uint64_t last_ip; -uint64_t last_dp; -uint8_t xmm[16 ][16 ]; -uint32_t mxcsr; -uint32_t pad2; -}MshvFPU ; typedef enum MshvVmExit { MshvVmExitIgnore = 0,
@@ -81,7 +68,7 @@ typedef enum MshvVmExit {void mshv_init_mmio_emu (void );int mshv_create_vcpu (int vm_fd,uint8_t vp_index,int *cpu_fd);void mshv_remove_vcpu (int vm_fd,int cpu_fd); -int mshv_configure_vcpu (const CPUState *cpu,const MshvFPU *fpu,uint64_t xcr0); +int mshv_configure_vcpu (const CPUState *cpu);int mshv_run_vcpu (int vm_fd,CPUState *cpu,hv_message *msg,MshvVmExit *exit);int mshv_arch_load_regs (CPUState *cpu);int mshv_arch_store_regs (CPUState *cpu);
target/i386/mshv/mshv-cpu.c +46 -30
160 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 3/34 target/i386/mshv: impl init/load/store_vcpu_state
In reply to Magnus KulkeIn migration we will handle more than registers, so we rework the routines that were used to load & store CPU registers from/to the hypervisor into more explicit init/load/store_vcpu_state() functions that can be called from the appropriate hooks.
load/store_regs() still exists for the purpose of MMIO emulation, but it will only address standard and special x86 registers.
Functions to retrieve FPU and XCR0 state from the hypervsisor have been introduced.
MSR and APIC state covered are covered only as part of init_vcpu(). They are not yet part of the load/store routines.
--- accel/mshv/mshv-all.c | 10 +- include/system/mshv_int.h | 5 +- target/i386/mshv/mshv-cpu.c | 354 ++++++++++++++++++------------------ 3 files changed, 185 insertions(+), 184 deletions(-)
accel/mshv/mshv-all.c +5 -5
@@ -400,13 +400,13 @@ static int mshv_init_vcpu(CPUState *cpu)int ret; cpu->accel =g_new0 (AccelCPUState,1 ); -mshv_arch_init_vcpu (cpu); ret =mshv_create_vcpu (vm_fd, vp_index, &cpu->accel->cpufd);if (ret <0 ) {return -1 ; } +mshv_arch_init_vcpu (cpu); cpu->accel->dirty = true;return 0 ;
@@ -488,7 +488,7 @@ static int mshv_cpu_exec(CPUState *cpu) do { if (cpu->accel->dirty) { - ret = mshv_arch_put_registers(cpu); + ret = mshv_arch_store_vcpu_state(cpu); if (ret) { error_report("Failed to put registers after init: %s", strerror(-ret));
@@ -610,7 +610,7 @@ static void mshv_start_vcpu_thread(CPUState *cpu)static void do_mshv_cpu_synchronize_post_init (CPUState *cpu,run_on_cpu_data arg) { -int ret =mshv_arch_put_registers (cpu); +int ret =mshv_arch_store_vcpu_state (cpu);if (ret <0 ) {error_report ("Failed to put registers after init: %s" ,strerror (-ret));abort ();
@@ -626,7 +626,7 @@ static void mshv_cpu_synchronize_post_init(CPUState *cpu)static void mshv_cpu_synchronize_post_reset (CPUState *cpu) { -int ret =mshv_arch_put_registers (cpu); +int ret =mshv_arch_store_vcpu_state (cpu);if (ret) {error_report ("Failed to put registers after reset: %s" ,strerror (-ret));
@@ -650,7 +650,7 @@ static void mshv_cpu_synchronize_pre_loadvm(CPUState *cpu)static void do_mshv_cpu_synchronize (CPUState *cpu,run_on_cpu_data arg) {if (!cpu->accel->dirty) { -int ret =mshv_arch_load_regs (cpu); +int ret =mshv_arch_load_vcpu_state (cpu);if (ret <0 ) {error_report ("Failed to load registers for vcpu %d" , cpu->cpu_index);
include/system/mshv_int.h +2 -3
@@ -70,11 +70,10 @@ int mshv_create_vcpu(int vm_fd, uint8_t vp_index, int *cpu_fd);void mshv_remove_vcpu (int vm_fd,int cpu_fd);int mshv_configure_vcpu (const CPUState *cpu);int mshv_run_vcpu (int vm_fd,CPUState *cpu,hv_message *msg,MshvVmExit *exit); -int mshv_arch_load_regs (CPUState *cpu); -int mshv_arch_store_regs (CPUState *cpu);int mshv_set_generic_regs (const CPUState *cpu,const hv_register_assoc *assocs,size_t n_regs); -int mshv_arch_put_registers (const CPUState *cpu); +int mshv_arch_store_vcpu_state (const CPUState *cpu); +int mshv_arch_load_vcpu_state (CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +178 -176
486 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 4/34 accel/accel-irq: add AccelRouteChange abstraction
In reply to Magnus KulkeThe accelerated irqchip routines use a record of changes to batch changes when programming routes.
Currently this mechanism is coupled to the KVM accelerator, this change introduces an abstraction that replaces KVMRouteChange and keeps a pointer to an abstract AccelState instead of the concrete type, converting the state where necessary.
This is done to further align the irqchip programming in the MSHV accelerator with the existing KVM code in QEMU. Subsequent commits will introduce AccelRouteChange to the MSHV accelerator code.
--- accel/accel-irq.c | 4 ++-- accel/kvm/kvm-all.c | 6 +++--- accel/stubs/kvm-stub.c | 2 +- hw/misc/ivshmem-pci.c | 2 +- hw/vfio/pci.c | 2 +- hw/virtio/virtio-pci.c | 3 +-- include/accel/accel-route.h | 17 +++++++++++++++++ include/system/accel-irq.h | 5 +++-- include/system/kvm.h | 21 ++++++++++----------- include/system/mshv.h | 1 + target/i386/kvm/kvm.c | 2 +- 11 files changed, 41 insertions(+), 24 deletions(-) create mode 100644 include/accel/accel-route.h
accel/accel-irq.c +2 -2
@@ -16,7 +16,7 @@#include "system/mshv.h" #include "system/accel-irq.h" -int accel_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev) +int accel_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev) {#ifdef CONFIG_MSHV_IS_POSSIBLE if (mshv_msi_via_irqfd_enabled ()) {
@@ -42,7 +42,7 @@ int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev)return -ENOSYS ; } -void accel_irqchip_commit_route_changes (KVMRouteChange *c) +void accel_irqchip_commit_route_changes (AccelRouteChange *c) {#ifdef CONFIG_MSHV_IS_POSSIBLE if (mshv_msi_via_irqfd_enabled ()) {
accel/kvm/kvm-all.c +3 -3
@@ -2359,11 +2359,11 @@ int kvm_irqchip_send_msi(KVMState *s, MSIMessage msg)return kvm_vm_ioctl (s,KVM_SIGNAL_MSI , &msi); } -int kvm_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev) +int kvm_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev) {struct kvm_irq_routing_entry kroute = {};int virq; -KVMState *s = c->s; +KVMState *s =KVM_STATE (c->accel);MSIMessage msg = {0 ,0 };if (pci_available && dev) {
@@ -2506,7 +2506,7 @@ int kvm_irqchip_send_msi(KVMState *s, MSIMessage msg)abort (); } -int kvm_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev) +int kvm_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev) {return -ENOSYS ; }
accel/stubs/kvm-stub.c +1 -1
@@ -44,7 +44,7 @@ int kvm_on_sigbus(int code, void *addr)return 1 ; } -int kvm_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev) +int kvm_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev) {return -ENOSYS ; }
hw/misc/ivshmem-pci.c +1 -1
@@ -424,7 +424,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector, Error **errp) {PCIDevice *pdev =PCI_DEVICE (s); -KVMRouteChange c; +AccelRouteChange c;int ret;IVSHMEM_DPRINTF ("ivshmem_add_kvm_msi_virq vector:%d\n" , vector);
hw/vfio/pci.c +1 -1
@@ -51,7 +51,7 @@#include "vfio-helpers.h" /* Protected by BQL */ -static KVMRouteChange vfio_route_change; +static AccelRouteChange vfio_route_change;static void vfio_disable_interrupts (VFIOPCIDevice *vdev);static void vfio_mmap_set_enabled (VFIOPCIDevice *vdev,bool enabled);
hw/virtio/virtio-pci.c +1 -2
@@ -869,7 +869,7 @@ static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,int ret;if (irqfd->users ==0 ) { -KVMRouteChange c =kvm_irqchip_begin_route_changes (kvm_state); +AccelRouteChange c =kvm_irqchip_begin_route_changes (kvm_state); ret =accel_irqchip_add_msi_route (&c, vector, &proxy->pci_dev);if (ret <0 ) {return ret;
@@ -2695,4 +2695,3 @@ static void virtio_pci_register_types(void) }type_init (virtio_pci_register_types) -
include/accel/accel-route.h new file +17 -0
@@ -0,0 +1,17 @@ +/* +* Accelerator MSI route change tracking +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#ifndef ACCEL_ROUTE_H +#define ACCEL_ROUTE_H + +#include "qemu/accel.h" + +typedef struct AccelRouteChange { +AccelState *accel; +int changes; +}AccelRouteChange ; + +#endif /* ACCEL_ROUTE_H */
include/system/accel-irq.h +3 -2
@@ -25,9 +25,10 @@ static inline bool accel_irqchip_is_split(void)return mshv_msi_via_irqfd_enabled () ||kvm_irqchip_is_split (); } -int accel_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev); +int accel_irqchip_add_msi_route (AccelRouteChange *c,int vector, +PCIDevice *dev);int accel_irqchip_update_msi_route (int vector,MSIMessage msg,PCIDevice *dev); -void accel_irqchip_commit_route_changes (KVMRouteChange *c); +void accel_irqchip_commit_route_changes (AccelRouteChange *c);void accel_irqchip_commit_routes (void );void accel_irqchip_release_virq (int virq); int accel_irqchip_add_irqfd_notifier_gsi(EventNotifier *n, EventNotifier *rn,
include/system/kvm.h +10 -11
@@ -18,6 +18,7 @@#include "exec/memattrs.h" #include "qemu/accel.h" +#include "accel/accel-route.h" #include "qom/object.h" #ifdef COMPILING_PER_TARGET
@@ -183,11 +184,6 @@ extern KVMState *kvm_state;typedef struct Notifier Notifier ;typedef struct NotifierWithReturn NotifierWithReturn ; -typedef struct KVMRouteChange { -KVMState *s; -int changes; -}KVMRouteChange ; -/* external API */ unsigned int kvm_get_max_memslots (void );
@@ -466,7 +462,7 @@ void kvm_init_cpu_signals(CPUState *cpu);/** * kvm_irqchip_add_msi_route - Add MSI route for specific vector -* @c: KVMRouteChange instance. +* @c: AccelRouteChange instance. * @vector: which vector to add. This can be either MSI/MSIX * vector. The function will automatically detect whether * MSI/MSIX is enabled, and fetch corresponding MSI
@@ -475,20 +471,23 @@ void kvm_init_cpu_signals(CPUState *cpu);* as @NULL, an empty MSI message will be inited. * @return: virq (>=0) when success, errno (<0) when failed. */ -int kvm_irqchip_add_msi_route (KVMRouteChange *c,int vector,PCIDevice *dev); +int kvm_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev);int kvm_irqchip_update_msi_route (KVMState *s,int virq,MSIMessage msg,PCIDevice *dev);void kvm_irqchip_commit_routes (KVMState *s); -static inline KVMRouteChange kvm_irqchip_begin_route_changes (KVMState *s) +static inline AccelRouteChange kvm_irqchip_begin_route_changes (KVMState *s) { -return (KVMRouteChange ) { .s = s, .changes =0 }; +return (AccelRouteChange ) { + .accel =ACCEL (s), + .changes =0 , + }; } -static inline void kvm_irqchip_commit_route_changes (KVMRouteChange *c) +static inline void kvm_irqchip_commit_route_changes (AccelRouteChange *c) {if (c->changes) { -kvm_irqchip_commit_routes (c->s); +kvm_irqchip_commit_routes (KVM_STATE (c->accel)); c->changes =0 ; } }
include/system/mshv.h +1 -0
@@ -21,6 +21,7 @@#include "qapi/qapi-types-common.h" #include "system/memory.h" #include "accel/accel-ops.h" +#include "accel/accel-route.h" #ifdef COMPILING_PER_TARGET #ifdef CONFIG_MSHV
target/i386/kvm/kvm.c +1 -1
@@ -6678,7 +6678,7 @@ void kvm_arch_init_irq_routing(KVMState *s) kvm_gsi_routing_allowed = true;if (kvm_irqchip_is_split ()) { -KVMRouteChange c =kvm_irqchip_begin_route_changes (s); +AccelRouteChange c =kvm_irqchip_begin_route_changes (s);int i;/* If the ioapic is in QEMU and the lapics are in KVM, reserve
Signature · 1 line
2.34.1
PATCH 5/34 accel/accel-irq: add generic begin_route_changes
In reply to Magnus KulkeA generic accel_irqchip_begin_route_change() fn has been introduced for usage in the MSHV accelerator. It replaces the respective kvm_ fn.
--- accel/accel-irq.c | 21 +++++++++++++++++++++ hw/misc/ivshmem-pci.c | 4 ++-- hw/vfio/pci.c | 5 +++-- hw/virtio/virtio-pci.c | 2 +- include/system/accel-irq.h | 1 + include/system/kvm.h | 8 -------- target/i386/kvm/kvm.c | 3 ++- 7 files changed, 30 insertions(+), 14 deletions(-)
accel/accel-irq.c +21 -0
@@ -10,6 +10,7 @@*/ #include "qemu/osdep.h" +#include "qemu/error-report.h" #include "hw/pci/msi.h" #include "system/kvm.h"
@@ -104,3 +105,23 @@ int accel_irqchip_remove_irqfd_notifier_gsi(EventNotifier *n, int virq) }return -ENOSYS ; } + +inline AccelRouteChange accel_irqchip_begin_route_changes (void ) +{ +#ifdef CONFIG_MSHV_IS_POSSIBLE +if (mshv_msi_via_irqfd_enabled ()) { +return (AccelRouteChange ) { + .accel =ACCEL (mshv_state), + .changes =0 , + }; + } +#endif +if (kvm_enabled ()) { +return (AccelRouteChange ) { + .accel =ACCEL (kvm_state), + .changes =0 , + }; + } +error_report ("can't initiate route change, no accel irqchip available" ); +abort (); +}
hw/misc/ivshmem-pci.c +2 -2
@@ -26,7 +26,7 @@#include "hw/core/qdev-properties-system.h" #include "hw/pci/msi.h" #include "hw/pci/msix.h" -#include "system/kvm.h" +#include "system/accel-irq.h" #include "migration/blocker.h" #include "migration/vmstate.h" #include "qemu/error-report.h"
@@ -430,7 +430,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector,IVSHMEM_DPRINTF ("ivshmem_add_kvm_msi_virq vector:%d\n" , vector);assert (!s->msi_vectors[vector].pdev); - c =kvm_irqchip_begin_route_changes (kvm_state); + c =accel_irqchip_begin_route_changes (); ret =kvm_irqchip_add_msi_route (&c, vector, pdev);if (ret <0 ) {error_setg (errp,"kvm_irqchip_add_msi_route failed" );
hw/vfio/pci.c +3 -2
@@ -39,6 +39,7 @@#include "qemu/module.h" #include "qemu/range.h" #include "qemu/units.h" +#include "system/accel-irq.h" #include "system/kvm.h" #include "system/runstate.h" #include "pci.h"
@@ -692,7 +693,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,if (vdev->defer_kvm_irq_routing) {vfio_pci_add_kvm_msi_virq (vdev, vector, nr, true); }else { - vfio_route_change =kvm_irqchip_begin_route_changes (kvm_state); + vfio_route_change =accel_irqchip_begin_route_changes ();vfio_pci_add_kvm_msi_virq (vdev, vector, nr, true);kvm_irqchip_commit_route_changes (&vfio_route_change);vfio_connect_kvm_msi_virq (vector, nr);
@@ -793,7 +794,7 @@ void vfio_pci_prepare_kvm_msi_virq_batch(VFIOPCIDevice *vdev) {assert (!vdev->defer_kvm_irq_routing); vdev->defer_kvm_irq_routing = true; - vfio_route_change =kvm_irqchip_begin_route_changes (kvm_state); + vfio_route_change =accel_irqchip_begin_route_changes (); }void vfio_pci_commit_kvm_msi_virq_batch (VFIOPCIDevice *vdev)
hw/virtio/virtio-pci.c +1 -1
@@ -869,7 +869,7 @@ static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,int ret;if (irqfd->users ==0 ) { -AccelRouteChange c =kvm_irqchip_begin_route_changes (kvm_state); +AccelRouteChange c =accel_irqchip_begin_route_changes (); ret =accel_irqchip_add_msi_route (&c, vector, &proxy->pci_dev);if (ret <0 ) {return ret;
include/system/accel-irq.h +1 -0
@@ -25,6 +25,7 @@ static inline bool accel_irqchip_is_split(void)return mshv_msi_via_irqfd_enabled () ||kvm_irqchip_is_split (); } +AccelRouteChange accel_irqchip_begin_route_changes (void );int accel_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev);int accel_irqchip_update_msi_route (int vector,MSIMessage msg,PCIDevice *dev);
include/system/kvm.h +0 -8
@@ -476,14 +476,6 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg, PCIDevice *dev);void kvm_irqchip_commit_routes (KVMState *s); -static inline AccelRouteChange kvm_irqchip_begin_route_changes (KVMState *s) -{ -return (AccelRouteChange ) { - .accel =ACCEL (s), - .changes =0 , - }; -} -static inline void kvm_irqchip_commit_route_changes (AccelRouteChange *c) {if (c->changes) {
target/i386/kvm/kvm.c +2 -1
@@ -32,6 +32,7 @@#include "vmsr_energy.h" #include "system/system.h" #include "system/hw_accel.h" +#include "system/accel-irq.h" #include "system/kvm_int.h" #include "system/runstate.h" #include "system/ramblock.h"
@@ -6678,7 +6679,7 @@ void kvm_arch_init_irq_routing(KVMState *s) kvm_gsi_routing_allowed = true;if (kvm_irqchip_is_split ()) { -AccelRouteChange c =kvm_irqchip_begin_route_changes (s); +AccelRouteChange c =accel_irqchip_begin_route_changes ();int i;/* If the ioapic is in QEMU and the lapics are in KVM, reserve
Signature · 1 line
2.34.1
PATCH 6/34 accel/accel-irq: add generic commit_route_changes
In reply to Magnus KulkeA generic accel_irqchip_commit_route_changes() fn has been introduced for usage in the MSHV accelerator. The respective kvm_ fn can be removed since we handle the commit op in a generic way.
--- accel/accel-irq.c | 10 +++------- hw/misc/ivshmem-pci.c | 2 +- hw/vfio/pci.c | 4 ++-- include/system/kvm.h | 8 -------- target/i386/kvm/kvm.c | 2 +- 5 files changed, 7 insertions(+), 19 deletions(-)
accel/accel-irq.c +3 -7
@@ -45,13 +45,9 @@ int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev)void accel_irqchip_commit_route_changes (AccelRouteChange *c) { -#ifdef CONFIG_MSHV_IS_POSSIBLE -if (mshv_msi_via_irqfd_enabled ()) { -mshv_irqchip_commit_routes (); - } -#endif -if (kvm_enabled ()) { -kvm_irqchip_commit_route_changes (c); +if (c->changes) { +accel_irqchip_commit_routes (); + c->changes =0 ; } }
hw/misc/ivshmem-pci.c +1 -1
@@ -436,7 +436,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector,error_setg (errp,"kvm_irqchip_add_msi_route failed" );return ; } -kvm_irqchip_commit_route_changes (&c); +accel_irqchip_commit_route_changes (&c); s->msi_vectors[vector].virq = ret; s->msi_vectors[vector].pdev = pdev;
hw/vfio/pci.c +2 -2
@@ -695,7 +695,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, }else { vfio_route_change =accel_irqchip_begin_route_changes ();vfio_pci_add_kvm_msi_virq (vdev, vector, nr, true); -kvm_irqchip_commit_route_changes (&vfio_route_change); +accel_irqchip_commit_route_changes (&vfio_route_change);vfio_connect_kvm_msi_virq (vector, nr); } }
@@ -804,7 +804,7 @@ void vfio_pci_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev)assert (vdev->defer_kvm_irq_routing); vdev->defer_kvm_irq_routing = false; -kvm_irqchip_commit_route_changes (&vfio_route_change); +accel_irqchip_commit_route_changes (&vfio_route_change);for (i =0 ; i < vdev->nr_vectors; i++) {vfio_connect_kvm_msi_virq (&vdev->msi_vectors[i], i);
include/system/kvm.h +0 -8
@@ -476,14 +476,6 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg, PCIDevice *dev);void kvm_irqchip_commit_routes (KVMState *s); -static inline void kvm_irqchip_commit_route_changes (AccelRouteChange *c) -{ -if (c->changes) { -kvm_irqchip_commit_routes (KVM_STATE (c->accel)); - c->changes =0 ; - } -} -int kvm_irqchip_get_virq (KVMState *s);void kvm_irqchip_release_virq (KVMState *s,int virq);
target/i386/kvm/kvm.c +1 -1
@@ -6690,7 +6690,7 @@ void kvm_arch_init_irq_routing(KVMState *s)exit (1 ); } } -kvm_irqchip_commit_route_changes (&c); +accel_irqchip_commit_route_changes (&c); } }
Signature · 1 line
2.34.1
PATCH 7/34 accel/mshv: add irq_routes to state
In reply to Magnus KulkeThis change adds fields related to irq routing to the MSHV state, following similar fields in the KVM implementation.
So far the fields are only initialized, they will be used in subsequent commits for bookkeeping purposes and storing uncommitted interrupt routes.
The TYPE_MSHV_ACCEL defines have been moved to the header.
--- accel/mshv/irq.c | 10 ++++++++++ accel/mshv/mshv-all.c | 6 ++---- include/system/mshv.h | 7 +++++++ include/system/mshv_int.h | 5 +++++ 4 files changed, 24 insertions(+), 4 deletions(-)
accel/mshv/irq.c +10 -0
@@ -396,3 +396,13 @@ int mshv_reserve_ioapic_msi_routes(int vm_fd)return 0 ; } + +void mshv_init_irq_routing (MshvState *s) +{ +int gsi_count =MSHV_MAX_MSI_ROUTES ; + + s->irq_routes =g_malloc0 (sizeof (*s->irq_routes)); + s->nr_allocated_irq_routes =0 ; + s->gsi_count = gsi_count; + s->used_gsi_bitmap =bitmap_new (gsi_count); +}
accel/mshv/mshv-all.c +2 -4
@@ -43,10 +43,6 @@#include <err.h> #include <sys/ioctl.h> -#define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv") - -DECLARE_INSTANCE_CHECKER(MshvState, MSHV_STATE, TYPE_MSHV_ACCEL) -bool mshv_allowed;MshvState *mshv_state;
@@ -457,6 +453,8 @@ static int mshv_init(AccelState *as, MachineState *ms) mshv_state = s; +mshv_init_irq_routing (s); +register_mshv_memory_listener (s, &s->memory_listener, &address_space_memory,0 ,"mshv-memory" );memory_listener_register (&mshv_io_listener, &address_space_io);
include/system/mshv.h +7 -0
@@ -45,7 +45,13 @@ extern bool mshv_allowed;#define mshv_msi_via_irqfd_enabled () mshv_enabled()#endif +#define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv") +typedef struct MshvState MshvState ; + +DECLARE_INSTANCE_CHECKER (MshvState,MSHV_STATE , + TYPE_MSHV_ACCEL) +extern MshvState *mshv_state;/* interrupt */
@@ -60,5 +66,6 @@ void mshv_irqchip_release_virq(int virq);int mshv_irqchip_add_irqfd_notifier_gsi (const EventNotifier *n,const EventNotifier *rn,int virq);int mshv_irqchip_remove_irqfd_notifier_gsi (const EventNotifier *n,int virq); +void mshv_init_irq_routing (MshvState *s);#endif
include/system/mshv_int.h +5 -0
@@ -48,6 +48,11 @@ struct MshvState {int nr_as;MshvAddressSpace *as;int fd; +/* irqchip routing */ +struct mshv_user_irq_table *irq_routes; +int nr_allocated_irq_routes; +unsigned long *used_gsi_bitmap; +unsigned int gsi_count; }; typedef struct MshvMsiControl {
Signature · 1 line
2.34.1
PATCH 8/34 accel/mshv: update s->irq_routes in add_msi_route
In reply to Magnus KulkeThe irq_routes field of the state is populated with native mshv irq route entries. The allocation logic is modelled after the KVM
implementation: we will always allocate a minumum of 64 entries and use a bitmask to find/set/clear GSIs.
The old implementation of add_msi_routes will be removed in a subsequent commit.
--- accel/accel-irq.c | 2 +- accel/mshv/irq.c | 87 +++++++++++++++++++++++++++++++++++++---- accel/stubs/mshv-stub.c | 2 +- include/system/mshv.h | 3 +- 4 files changed, 84 insertions(+), 10 deletions(-)
accel/accel-irq.c +1 -1
@@ -21,7 +21,7 @@ int accel_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev) {#ifdef CONFIG_MSHV_IS_POSSIBLE if (mshv_msi_via_irqfd_enabled ()) { -return mshv_irqchip_add_msi_route (vector, dev); +return mshv_irqchip_add_msi_route (c, vector, dev); }#endif if (kvm_enabled ()) {
accel/mshv/irq.c +80 -7
98 lines not on this page · raw message
accel/stubs/mshv-stub.c +1 -1
@@ -14,7 +14,7 @@bool mshv_allowed; -int mshv_irqchip_add_msi_route (int vector,PCIDevice *dev) +int mshv_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev) {return -ENOSYS ; }
include/system/mshv.h +2 -1
@@ -33,6 +33,7 @@#endif #define MSHV_MAX_MSI_ROUTES 4096 +#define MSHV_MIN_ALLOCATED_MSI_ROUTES 64#define MSHV_PAGE_SHIFT 12
@@ -59,7 +60,7 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3 uint32_t vp_index, bool logical_destination_mode, bool level_triggered); -int mshv_irqchip_add_msi_route (int vector,PCIDevice *dev); +int mshv_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev);int mshv_irqchip_update_msi_route (int virq,MSIMessage msg,PCIDevice *dev);void mshv_irqchip_commit_routes (void );void mshv_irqchip_release_virq (int virq);
Signature · 1 line
2.34.1
PATCH 9/34 accel/mshv: update s->irq_routes in update_msi_route
In reply to Magnus KulkeThe state's irq_routes field will be updated when an irqchip's gsi is requested to be updated with a new dest/vector.
The old set_msi_routing() fn is redundant and can be removed.
--- accel/mshv/irq.c | 87 +++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 49 deletions(-)
accel/mshv/irq.c +38 -49
@@ -36,52 +36,6 @@ void mshv_init_msicontrol(void) msi_control->updated = false; } -static int set_msi_routing (uint32_t gsi,uint64_t addr,uint32_t data) -{ -struct mshv_user_irq_entry *entry; -uint32_t high_addr = addr >>32 ; -uint32_t low_addr = addr &0xFFFFFFFF ; -GHashTable *gsi_routes; - -trace_mshv_set_msi_routing (gsi, addr, data); - -if (gsi >=MSHV_MAX_MSI_ROUTES ) { -error_report ("gsi >= MSHV_MAX_MSI_ROUTES" ); -return -1 ; - } - -assert (msi_control); - - WITH_QEMU_LOCK_GUARD(&msi_control_mutex) { - gsi_routes = msi_control->gsi_routes; - entry =g_hash_table_lookup (gsi_routes,GINT_TO_POINTER (gsi)); - -if (entry - && entry->address_hi == high_addr - && entry->address_lo == low_addr - && entry->data == data) - { -/* nothing to update */ -return 0 ; - } - -/* free old entry */ -g_free (entry); - -/* create new entry */ - entry = g_new0(struct mshv_user_irq_entry, 1); - entry->gsi = gsi; - entry->address_hi = high_addr; - entry->address_lo = low_addr; - entry->data = data; - -g_hash_table_insert (gsi_routes,GINT_TO_POINTER (gsi), entry); - msi_control->updated = true; - } - -return 0 ; -} -static int add_msi_routing (uint64_t addr,uint32_t data) {struct mshv_user_irq_entry *route_entry;
@@ -370,16 +324,51 @@ void mshv_irqchip_release_virq(int virq)remove_msi_routing (virq); } +static int update_routing_entry (MshvState *s, +struct mshv_user_irq_entry *new_entry) +{ +struct mshv_user_irq_entry *entry; +int n; + +for (n =0 ; n < s->irq_routes->nr; n++) { + entry = &s->irq_routes->entries[n]; +if (entry->gsi != new_entry->gsi) { +continue ; + } + +if (!memcmp (entry, new_entry,sizeof *entry)) { +return 0 ; + } + + *entry = *new_entry; + +return 0 ; + } + +return -ESRCH ; +} +int mshv_irqchip_update_msi_route (int virq,MSIMessage msg,PCIDevice *dev) { +uint32_t addr_hi = msg.address >>32 ; +uint32_t addr_lo = msg.address &0xFFFFFFFF ; +uint32_t data =le32_to_cpu (msg.data); +struct mshv_user_irq_entry entry = { + .gsi = virq, + .address_hi = addr_hi, + .address_lo = addr_lo, + .data = data, + };int ret; - ret =set_msi_routing (virq, msg.address,le32_to_cpu (msg.data)); + ret =update_routing_entry (mshv_state, &entry);if (ret <0 ) { -error_report ("Failed to set msi routing" ); -return -1 ; +error_report ("Failed to set msi routing for gsi %d" , virq); +abort (); } +trace_mshv_set_msi_routing (virq, msg.address, data); +return 0 ; }
Signature · 1 line
2.34.1
PATCH 10/34 accel/mshv: update s->irq_routes in release_virq
In reply to Magnus KulkeThe state's irq_routes field will be updated when an irqchip's gsi is requested to be released.
The old remove_msi_routing() fn is redundant and can be removed.
--- accel/accel-irq.c | 2 +- accel/mshv/irq.c | 43 ++++++++++++++--------------------------- accel/stubs/mshv-stub.c | 2 +- include/system/mshv.h | 2 +- 4 files changed, 17 insertions(+), 32 deletions(-)
accel/accel-irq.c +1 -1
@@ -67,7 +67,7 @@ void accel_irqchip_release_virq(int virq) {#ifdef CONFIG_MSHV_IS_POSSIBLE if (mshv_msi_via_irqfd_enabled ()) { -mshv_irqchip_release_virq (virq); +mshv_irqchip_release_virq (mshv_state, virq); }#endif if (kvm_enabled ()) {
accel/mshv/irq.c +14 -29
56 lines not on this page · raw message
accel/stubs/mshv-stub.c +1 -1
@@ -19,7 +19,7 @@ int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)return -ENOSYS ; } -void mshv_irqchip_release_virq (int virq) +void mshv_irqchip_release_virq (MshvState *s,int virq) { }
include/system/mshv.h +1 -1
@@ -63,7 +63,7 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3int mshv_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev);int mshv_irqchip_update_msi_route (int virq,MSIMessage msg,PCIDevice *dev);void mshv_irqchip_commit_routes (void ); -void mshv_irqchip_release_virq (int virq); +void mshv_irqchip_release_virq (MshvState *s,int virq);int mshv_irqchip_add_irqfd_notifier_gsi (const EventNotifier *n,const EventNotifier *rn,int virq);int mshv_irqchip_remove_irqfd_notifier_gsi (const EventNotifier *n,int virq);
Signature · 1 line
2.34.1
PATCH 11/34 accel/mshv: use s->irq_routes in commit_routes
In reply to Magnus KulkeIn mshv_irqchip_commit_routes() the entries that have been accumulated in s->irq_routes are committed directly to MSHV's irqchip.
The old commit_msi_routing_table() fn will be removed in a subsquent commit.
--- accel/accel-irq.c | 2 +- accel/mshv/irq.c | 7 ++++--- accel/stubs/mshv-stub.c | 2 +- include/system/mshv.h | 2 +- 4 files changed, 7 insertions(+), 6 deletions(-)
accel/accel-irq.c +1 -1
@@ -55,7 +55,7 @@ void accel_irqchip_commit_routes(void) {#ifdef CONFIG_MSHV_IS_POSSIBLE if (mshv_msi_via_irqfd_enabled ()) { -mshv_irqchip_commit_routes (); +mshv_irqchip_commit_routes (mshv_state); }#endif if (kvm_enabled ()) {
accel/mshv/irq.c +4 -3
@@ -394,16 +394,17 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3return 0 ; } -void mshv_irqchip_commit_routes (void ) +void mshv_irqchip_commit_routes (MshvState *s) {int ret; -int vm_fd = mshv_state->vm; +int vm_fd = s->vm; - ret =commit_msi_routing_table (vm_fd); + ret =ioctl (vm_fd,MSHV_SET_MSI_ROUTING , s->irq_routes);if (ret <0 ) {error_report ("Failed to commit msi routing table" );abort (); } +trace_mshv_commit_msi_routing_table (vm_fd, s->irq_routes->nr); } int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *event,
accel/stubs/mshv-stub.c +1 -1
@@ -28,7 +28,7 @@ int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev)return -ENOSYS ; } -void mshv_irqchip_commit_routes (void ) +void mshv_irqchip_commit_routes (MshvState *s) { }
include/system/mshv.h +1 -1
@@ -62,8 +62,8 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3int mshv_irqchip_add_msi_route (AccelRouteChange *c,int vector,PCIDevice *dev);int mshv_irqchip_update_msi_route (int virq,MSIMessage msg,PCIDevice *dev); -void mshv_irqchip_commit_routes (void );void mshv_irqchip_release_virq (MshvState *s,int virq); +void mshv_irqchip_commit_routes (MshvState *s);int mshv_irqchip_add_irqfd_notifier_gsi (const EventNotifier *n,const EventNotifier *rn,int virq);int mshv_irqchip_remove_irqfd_notifier_gsi (const EventNotifier *n,int virq);
Signature · 1 line
2.34.1
PATCH 12/34 accel/mshv: reserve ioapic routes on s->irq_routes
In reply to Magnus KulkeWe reserve 24 ioapic routes using the new functions that operate on the mshv apic state.
commit/add_msi_routing() fn's can be removed now.
--- accel/mshv/irq.c | 115 ++++++-------------------------------- accel/mshv/mshv-all.c | 5 -- include/system/mshv_int.h | 1 - 3 files changed, 18 insertions(+), 103 deletions(-)
accel/mshv/irq.c +18 -97
@@ -36,93 +36,6 @@ void mshv_init_msicontrol(void) msi_control->updated = false; } -static int add_msi_routing (uint64_t addr,uint32_t data) -{ -struct mshv_user_irq_entry *route_entry; -uint32_t high_addr = addr >>32 ; -uint32_t low_addr = addr &0xFFFFFFFF ; -int gsi; -GHashTable *gsi_routes; - -trace_mshv_add_msi_routing (addr, data); - -assert (msi_control); - - WITH_QEMU_LOCK_GUARD(&msi_control_mutex) { -/* find an empty slot */ - gsi =0 ; - gsi_routes = msi_control->gsi_routes; -while (gsi <MSHV_MAX_MSI_ROUTES ) { - route_entry =g_hash_table_lookup (gsi_routes,GINT_TO_POINTER (gsi)); -if (!route_entry) { -break ; - } - gsi++; - } -if (gsi >=MSHV_MAX_MSI_ROUTES ) { -error_report ("No empty gsi slot available" ); -return -1 ; - } - -/* create new entry */ - route_entry = g_new0(struct mshv_user_irq_entry, 1); - route_entry->gsi = gsi; - route_entry->address_hi = high_addr; - route_entry->address_lo = low_addr; - route_entry->data = data; - -g_hash_table_insert (gsi_routes,GINT_TO_POINTER (gsi), route_entry); - msi_control->updated = true; - } - -return gsi; -} - -static int commit_msi_routing_table (int vm_fd) -{ -guint len; -int i, ret; -size_t table_size; -struct mshv_user_irq_table *table; -GHashTableIter iter; -gpointer key, value; - -assert (msi_control); - - WITH_QEMU_LOCK_GUARD(&msi_control_mutex) { -if (!msi_control->updated) { -/* nothing to update */ -return 0 ; - } - -/* Calculate the size of the table */ - len =g_hash_table_size (msi_control->gsi_routes); - table_size =sizeof (struct mshv_user_irq_table ) - + len *sizeof (struct mshv_user_irq_entry ); - table =g_malloc0 (table_size); - -g_hash_table_iter_init (&iter, msi_control->gsi_routes); - i =0 ; -while (g_hash_table_iter_next (&iter, &key, &value)) { -struct mshv_user_irq_entry *entry = value; - table->entries[i] = *entry; - i++; - } - table->nr = i; - -trace_mshv_commit_msi_routing_table (vm_fd, len); - - ret =ioctl (vm_fd,MSHV_SET_MSI_ROUTING , table); -g_free (table); -if (ret <0 ) { -error_report ("Failed to commit msi routing table" ); -return -1 ; - } - msi_control->updated = false; - } -return 0 ; -} -/* Pass an eventfd which is to be used for injecting interrupts from userland */ static int irqfd (int vm_fd,int fd,int resample_fd,uint32_t gsi,uint32_t flags)
@@ -420,37 +333,45 @@ int mshv_irqchip_remove_irqfd_notifier_gsi(const EventNotifier *event,return irqchip_update_irqfd_notifier_gsi (event,NULL , virq, false); } -int mshv_reserve_ioapic_msi_routes (int vm_fd) +static int mshv_reserve_ioapic_msi_routes (MshvState *s) { -int ret, gsi; +int ret, i; +int gsi =0 ; +struct mshv_user_irq_entry blank_entry = {0 };/* * Reserve GSI 0-23 for IOAPIC pins, to avoid conflicts of legacy * peripherals with MSI-X devices */ -for (gsi =0 ; gsi <IOAPIC_NUM_PINS ; gsi++) { - ret =add_msi_routing (0 ,0 ); +for (i =0 ; i <IOAPIC_NUM_PINS ; i++) { +/* ret = add_msi_routing(0, 0); */ + ret =irqchip_allocate_gsi (s, &gsi);if (ret <0 ) { -error_report ("Failed to reserve GSI %d" , gsi); +error_report ("Failed to reserve GSI %d: %s" , gsi,strerror (-ret));return -1 ; } + blank_entry.gsi = gsi; +add_routing_entry (s, &blank_entry); } - ret =commit_msi_routing_table (vm_fd); -if (ret <0 ) { -error_report ("Failed to commit reserved IOAPIC MSI routes" ); -return -1 ; - } +mshv_irqchip_commit_routes (s);return 0 ; }void mshv_init_irq_routing (MshvState *s) { +int ret;int gsi_count =MSHV_MAX_MSI_ROUTES ; s->irq_routes =g_malloc0 (sizeof (*s->irq_routes)); s->nr_allocated_irq_routes =0 ; s->gsi_count = gsi_count; s->used_gsi_bitmap =bitmap_new (gsi_count); + + ret =mshv_reserve_ioapic_msi_routes (s); +if (ret <0 ) { +error_report ("Failed to reserve IOAPIC MSI routes" ); +abort (); + } }
accel/mshv/mshv-all.c +0 -5
@@ -200,11 +200,6 @@ static int create_vm(int mshv_fd, int *vm_fd)return -1 ; } - ret =mshv_reserve_ioapic_msi_routes (*vm_fd); -if (ret <0 ) { -return -1 ; - } - ret =mshv_arch_post_init_vm (*vm_fd);if (ret <0 ) {return -1 ;
include/system/mshv_int.h +0 -1
@@ -120,6 +120,5 @@ int mshv_configure_msr(const CPUState *cpu, const MshvMsrEntry *msrs,/* interrupt */ void mshv_init_msicontrol (void ); -int mshv_reserve_ioapic_msi_routes (int vm_fd);#endif
Signature · 1 line
2.34.1
PATCH 13/34 accel/mshv: remove redundant msi controller
In reply to Magnus KulkeThe remaining MsiControl infrastructure can be removed now
--- accel/mshv/irq.c | 11 ----------- accel/mshv/mshv-all.c | 2 -- include/system/mshv_int.h | 3 --- 3 files changed, 16 deletions(-)
accel/mshv/irq.c +0 -11
@@ -25,17 +25,6 @@#define MSHV_IRQFD_RESAMPLE_FLAG (1 << MSHV_IRQFD_BIT_RESAMPLE)#define MSHV_IRQFD_BIT_DEASSIGN_FLAG (1 << MSHV_IRQFD_BIT_DEASSIGN) -static MshvMsiControl *msi_control; -static QemuMutex msi_control_mutex; - -void mshv_init_msicontrol (void ) -{ -qemu_mutex_init (&msi_control_mutex); - msi_control =g_new0 (MshvMsiControl,1 ); - msi_control->gsi_routes =g_hash_table_new (g_direct_hash, g_direct_equal); - msi_control->updated = false; -} -/* Pass an eventfd which is to be used for injecting interrupts from userland */ static int irqfd (int vm_fd,int fd,int resample_fd,uint32_t gsi,uint32_t flags)
accel/mshv/mshv-all.c +0 -2
@@ -426,8 +426,6 @@ static int mshv_init(AccelState *as, MachineState *ms)mshv_init_mmio_emu (); -mshv_init_msicontrol (); - ret =create_vm (mshv_fd, &vm_fd);if (ret <0 ) {close (mshv_fd);
include/system/mshv_int.h +0 -3
@@ -118,7 +118,4 @@ typedef struct MshvMsrEntries {int mshv_configure_msr (const CPUState *cpu,const MshvMsrEntry *msrs,size_t n_msrs); -/* interrupt */ -void mshv_init_msicontrol (void ); -#endif
Signature · 1 line
2.34.1
PATCH 14/34 target/i386/mshv: move apic logic into own file
In reply to Magnus KulkeUse a new file to unclutter the vcpu code gradually from APIC related logic.
--- include/system/mshv_int.h | 8 ++++ target/i386/mshv/meson.build | 1 + target/i386/mshv/mshv-apic.c | 78 ++++++++++++++++++++++++++++++++++++ 3 files changed, 87 insertions(+) create mode 100644 target/i386/mshv/mshv-apic.c
include/system/mshv_int.h +8 -0
@@ -16,6 +16,8 @@#define MSHV_MSR_ENTRIES_COUNT 64 +struct mshv_get_set_vp_state ; +typedef struct hyperv_message hv_message ; typedef struct MshvHvCallArgs {
@@ -84,6 +86,12 @@ void mshv_arch_destroy_vcpu(CPUState *cpu);void mshv_arch_amend_proc_features (union hv_partition_synthetic_processor_features *features);int mshv_arch_post_init_vm (int vm_fd); +int mshv_set_lapic (int cpu_fd, +const struct hv_local_interrupt_controller_state *state); +int mshv_get_lapic (int cpu_fd, +struct hv_local_interrupt_controller_state *state); +int mshv_get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state); +int mshv_set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state);typedef struct mshv_root_hvcall mshv_root_hvcall ;int mshv_hvcall (int fd,const mshv_root_hvcall *args);
target/i386/mshv/meson.build +1 -0
@@ -1,6 +1,7 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files( + 'mshv-apic.c', 'mshv-cpu.c', ))
target/i386/mshv/mshv-apic.c new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 15/34 target/i386/mshv: remove redundant apic helpers
In reply to Magnus KulkeThis change remove the now redundant apic helpers in mshv-cpu.c
--- target/i386/mshv/mshv-cpu.c | 88 ++----------------------------------- 1 file changed, 3 insertions(+), 85 deletions(-)
target/i386/mshv/mshv-cpu.c +3 -85
@@ -919,101 +919,19 @@ static int set_xc_reg(const CPUState *cpu)return 0 ; } -static int get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state) -{ -int ret; - - ret =ioctl (cpu_fd,MSHV_GET_VP_STATE , state); -if (ret <0 ) { -error_report ("failed to get partition state: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int get_lapic (const CPUState *cpu, -struct hv_local_interrupt_controller_state *state) -{ -int ret; -size_t size =4096 ; -/* buffer aligned to 4k, as *state requires that */ -void *buffer =qemu_memalign (size, size); -struct mshv_get_set_vp_state mshv_state = {0 }; -int cpu_fd =mshv_vcpufd (cpu); - - mshv_state.buf_ptr = (uint64_t ) buffer; - mshv_state.buf_sz = size; - mshv_state.type =MSHV_VP_STATE_LAPIC ; - - ret =get_vp_state (cpu_fd, &mshv_state); -if (ret ==0 ) { -memcpy (state, buffer,sizeof (*state)); - } -qemu_vfree (buffer); -if (ret <0 ) { -error_report ("failed to get lapic" ); -return -1 ; - } - -return 0 ; -} -static uint32_t set_apic_delivery_mode (uint32_t reg,uint32_t mode) {return ((reg) & ~0x700 ) | ((mode) <<8 ); } -static int set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state) -{ -int ret; - - ret =ioctl (cpu_fd,MSHV_SET_VP_STATE , state); -if (ret <0 ) { -error_report ("failed to set partition state: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int set_lapic (const CPUState *cpu, -const struct hv_local_interrupt_controller_state *state) -{ -int ret; -size_t size =4096 ; -/* buffer aligned to 4k, as *state requires that */ -void *buffer =qemu_memalign (size, size); -struct mshv_get_set_vp_state mshv_state = {0 }; -int cpu_fd =mshv_vcpufd (cpu); - -if (!state) { -error_report ("lapic state is NULL" ); -return -1 ; - } -memcpy (buffer, state,sizeof (*state)); - - mshv_state.buf_ptr = (uint64_t ) buffer; - mshv_state.buf_sz = size; - mshv_state.type =MSHV_VP_STATE_LAPIC ; - - ret =set_vp_state (cpu_fd, &mshv_state); -qemu_vfree (buffer); -if (ret <0 ) { -error_report ("failed to set lapic: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} -static int init_lint (const CPUState *cpu) {int ret;uint32_t *lvt_lint0, *lvt_lint1; +int cpu_fd =mshv_vcpufd (cpu);struct hv_local_interrupt_controller_state lapic_state = {0 }; - ret =get_lapic (cpu, &lapic_state); + ret =mshv_get_lapic (cpu_fd, &lapic_state);if (ret <0 ) {return ret; }
@@ -1026,7 +944,7 @@ static int init_lint(const CPUState *cpu)/* TODO: should we skip setting lapic if the values are the same? */ -return set_lapic (cpu, &lapic_state); +return mshv_set_lapic (cpu_fd, &lapic_state); }int mshv_arch_store_vcpu_state (const CPUState *cpu)
Signature · 1 line
2.34.1
PATCH 16/34 target/i386/mshv: migrate LAPIC state
In reply to Magnus KulkeThis change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.
An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare VMSTATE_BUFFER for the hv_lapic_state field.
In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.
--- hw/intc/apic_common.c | 3 ++ include/hw/i386/apic_internal.h | 5 +++ target/i386/mshv/mshv-cpu.c | 61 +++++++++++++++++++++++++++++++-- 3 files changed, 67 insertions(+), 2 deletions(-)
hw/intc/apic_common.c +3 -0
@@ -380,6 +380,9 @@ static const VMStateDescription vmstate_apic_common = {VMSTATE_INT64 (next_time, APICCommonState),VMSTATE_INT64 (timer_expiry, APICCommonState),/* open-coded timer state */ +#ifdef CONFIG_MSHV +VMSTATE_BUFFER (hv_lapic_state, APICCommonState), +#endifVMSTATE_END_OF_LIST () }, .subsections = (const VMStateDescription * const []) {
include/hw/i386/apic_internal.h +5 -0
@@ -23,6 +23,7 @@#include "cpu.h" #include "hw/i386/apic.h" +#include "hw/hyperv/hvgdk_mini.h" #include "system/memory.h" #include "qemu/timer.h" #include "target/i386/cpu-qom.h"
@@ -188,6 +189,10 @@ struct APICCommonState {DeviceState *vapic;hwaddr vapic_paddr;/* note: persistence via kvmvapic */ uint32_t extended_log_dest; + +#ifdef CONFIG_MSHV +uint8_t hv_lapic_state[sizeof (struct hv_local_interrupt_controller_state )]; +#endif }; typedef struct VAPICState {
target/i386/mshv/mshv-cpu.c +59 -2
93 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 15 lines · Magnus Kulke
This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.
An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare
VMSTATE_BUFFERfor thehv_lapic_statefield.
Quoted 5 lines
In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.
Signed-off-by Magnus Kulke <[email protected]>
Hi,
Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?
Or is that not a wanted goal?
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 19 lines · Mohamed Mediouni
Quoted 19 lines · Magnus Kulke
This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.
An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare
VMSTATE_BUFFERfor thehv_lapic_statefield.In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.
Quoted 9 lines
Quoted 1 line
Signed-off-by Magnus Kulke <[email protected]>Hi,
Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?
Or is that not a wanted goal?
yeah, for the time being it's a non-goal. there might a way to map hyperv lapic state onto existing state fields, but the mshv crates (which is the closest to a reference impl for mshv uapi) currently only expose the state as an opaque buffer.
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
In reply to Mohamed MediouniQuoted 19 lines · Mohamed Mediouni
Quoted 19 lines · Magnus Kulke
This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.
An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare
VMSTATE_BUFFERfor thehv_lapic_statefield.In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.
Quoted 9 lines
Quoted 1 line
Signed-off-by Magnus Kulke <[email protected]>Hi,
Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?
Or is that not a wanted goal?
as we discussed offline: there is mapping code in the mshv-ioctls crate and some prior art in WHPX that we can use as a reference. So we are to able to roundtrip the lapic state using generic field on APICCommonState and a few additional mshv-specific fields. This seems to work reliably in my migration tests.
I'll update this in the next iteration of this series. A KVM <=> MSHV migration path remains a non-goal, but it'll be a bit nicer to have the state transparent in the migration stream.
best,
magnus
Paolo Bonzini
maintainerRe: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
In reply to Magnus KulkeIl ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]> ha scritto:
Quoted 1 line
+#ifdef
CONFIG_MSHV
+ VMSTATE_BUFFER(hv_lapic_state, APICCommonState),
+#endif
This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.
DeviceState *vapic;
Quoted 8 lines
hwaddr
vapic_paddr; /* note: persistence via kvmvapic */uint32_textended_log_dest;include/hw/i386/apic_internal.h +3
-0+ +#ifdef CONFIG_MSHV + uint8_t hv_lapic_state[sizeof(struct
hv_local_interrupt_controller_state)];
+#endif
No, this is called common for a reason. You need to implement a subclass and handle migration in there.
Paolo
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 15 lines · Paolo Bonzini
Il ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]> ha scritto:
Quoted 1 line
+#ifdef
CONFIG_MSHV+
VMSTATE_BUFFER(hv_lapic_state, APICCommonState),+#endif
This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.
DeviceState *vapic;
Quoted 12 lines
Quoted 8 lines
hwaddr
vapic_paddr; /* note: persistence via kvmvapic */uint32_textended_log_dest;include/hw/i386/apic_internal.h +3
-0+ +#ifdef CONFIG_MSHV + uint8_t hv_lapic_state[sizeof(struct
hv_local_interrupt_controller_state)];
+#endifNo, this is called common for a reason. You need to implement a subclass and handle migration in there.
makes sense. I think I found a way to put the additional fields into a subclass. the apic_id is semantic, so we cannot use the auto-wiring dc->vmds state, but APIC_COMMON is doing the same. I'll send a revised patch soon.
thanks,
magnus
Paolo Bonzini
maintainerRe: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Il mer 1 lug 2026, 17:30 Magnus Kulke <[email protected]> ha scritto:
Quoted 30 lines · Paolo Bonzini
Quoted 1 line
Il ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]>
ha
Quoted 14 lines
scritto:
Quoted 1 line
+#ifdef
CONFIG_MSHV+
VMSTATE_BUFFER(hv_lapic_state, APICCommonState),+#endif
This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.
DeviceState *vapic;Quoted 12 lines
Quoted 8 lines
hwaddr
vapic_paddr; /* note: persistence via kvmvapic */uint32_textended_log_dest;include/hw/i386/apic_internal.h +3
-0+ +#ifdef CONFIG_MSHV + uint8_t hv_lapic_state[sizeof(struct
hv_local_interrupt_controller_state)];
+#endifNo, this is called common for a reason. You need to implement a subclass and handle migration in there.
Quoted 2 lines
makes sense. I think I found a way to put the additional fields into a subclass.
What are the additional fields that aren't covered (i.e. cannot be copied from hv_lapic_state to the struct and back) by the common APIC state?
Paolo
the apic_id is semantic, so we cannot use the auto-wiring
Quoted 9 lines
dc->vmdsstate, butAPIC_COMMONis doing the same. I'll send a revised patch
soon.thanks,
magnus
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 3 lines · Paolo Bonzini
What are the additional fields that aren't covered (i.e. cannot be copied from
hv_lapic_stateto the struct and back) by the common APIC state?
that would be those:
uint32_t apic_version;
uint32_t apic_lvt_lint0;
uint32_t apic_lvt_lint1;
uint32_t apic_lvt_cmci;
uint32_t apic_error_status;
uint32_t apic_counter_value;
uint32_t apic_remote_read;Reference struct is this: hv_local_interrupt_controller_state
https://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h
Notably we have to do this for lvt_lint0+1. We cannot use the respective fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,
pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()
For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion. Hence we roundtrip lint0/lint1 in a dedicated mshv struct.
best,
magnus
Paolo Bonzini
maintainerRe: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Il gio 2 lug 2026, 11:49 Magnus Kulke <[email protected]> ha scritto:
Quoted 16 lines · Paolo Bonzini
Quoted 4 lines
What are the additional fields that aren't covered (i.e. cannot be copied from
hv_lapic_stateto the struct and back) by the common APIC state?that would be those:
uint32_t apic_version; uint32_t apic_lvt_lint0; uint32_t apic_lvt_lint1; uint32_t apic_lvt_cmci; uint32_t apic_error_status; uint32_t apic_counter_value; uint32_t apic_remote_read;
Quoted 3 lines
Reference struct is this:
hv_local_interrupt_controller_statehttps://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h
For anything that's new but part of the standard APIC you can place them in common APIC state, just making migration state part of a subsection.
Notably we have to do this for lvt_lint0+1. We cannot use the respective
Quoted 7 lines
fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,
pic_irq_request()=>apic_accept_pic_intr()=>apic_deliver_pic_intr()For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion.
What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.
For LINT1, can you not inject an NMI either?
Thanks,
Paolo
Hence we roundtrip
Quoted 8 lines
lint0/lint1 in a dedicated mshv struct.
best,
magnus
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 33 lines · Paolo Bonzini
Il gio 2 lug 2026, 11:49 Magnus Kulke <[email protected]> ha scritto:
Quoted 3 lines · Paolo Bonzini
What are the additional fields that aren't covered (i.e. cannot be copied from
hv_lapic_stateto the struct and back) by the common APIC state?that would be those:
uint32_t apic_version; uint32_t apic_lvt_lint0; uint32_t apic_lvt_lint1; uint32_t apic_lvt_cmci; uint32_t apic_error_status; uint32_t apic_counter_value; uint32_t apic_remote_read;Reference struct is this:
hv_local_interrupt_controller_statehttps://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h
For anything that's new but part of the standard APIC you can place them in common APIC state, just making migration state part of a subsection.
Notably we have to do this for lvt_lint0+1. We cannot use the respective fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,
Quoted 6 lines
pic_irq_request()=>apic_accept_pic_intr()=>apic_deliver_pic_intr()For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion.
What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.
Hi,
For injecting LINT0, it’s possible on modern WHPX when using the hw irqchip
- starting from Server 2022/Windows 11 only - not before that point.
How it’s done is though cancelling the vCPU execution, wait until the vm exits as a result, set an interrupt window, run the vCPU again and then when it exits Manually injecting an event…
That’s unlike non-legacy interrupts where the request interrupts path actually works on Hyper-V.
Quoted 2 lines
For LINT1, can you not inject an NMI either?
The request interrupt path works at least to some extent on LINT1/NMIs unlike LINT0 but didn’t dig deeper to see if it has more issues…
I’m convinced that an Hyper-V specific structure is the wrong idea for this though...
Quoted 12 lines
Thanks,
Paolo
Hence we roundtrip
lint0/lint1 in a dedicated mshv struct.best,
magnus
Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state
Quoted 16 lines · Mohamed Mediouni
Quoted 2 lines · Paolo Bonzini
What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.
Hi,
For injecting LINT0, it’s possible on modern WHPX when using the hw irqchip
- starting from Server 2022/Windows 11 only - not before that point.How it’s done is though cancelling the vCPU execution, wait until the vm exits as a result, set an interrupt window, run the vCPU again and then when it exits Manually injecting an event…
That’s unlike non-legacy interrupts where the request interrupts path actually works on Hyper-V.
Quoted 1 line
Quoted 6 lines
Quoted 1 line
For LINT1, can you not inject an NMI either?
The request interrupt path works at least to some extent on LINT1/NMIs unlike LINT0 but didn’t dig deeper to see if it has more issues…
I’m convinced that an Hyper-V specific structure is the wrong idea for this though...
Thanks for chiming in Mohamed. Indeed, I think the culprit is that injecting legacy interrupts is currently not working as expected on MSHV either, so this behaviour is probably related to Microsoft Hypervisor itself.
When you refer to "cancelling the vCPU exection", I suspect you mean manipulating the intercept/explicit suspend registers? I understand that the mshv driver maintainers would prefer not to expose that particular knob, since the kernel driver maintains some internal suspend state. Thus the hvcall is blocked from dom0 userspace currently. I'll have to further clarify that with the mshv team, though.
Mirroring the LINT0/1 fields in a dedicated MsvhApic structure was motivated by the legacy interrupt injection path not working and thus breaking the destination on a migration. Assuming we don't have the option to make it work without a kernel change (in the short term), what would be your suggestion?
I can image we can short-circuit this chain for mshv, is that preferable?
pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()
best,
magnus
PATCH 17/34 target/i386/mshv: move msr code to arch
In reply to Magnus KulkeThe MSR code is x86 specific, hence it's better suited in the arch tree.
---
accel/mshv/meson.build | 1 -
target/i386/mshv/meson.build | 1 +
{accel => target/i386}/mshv/msr.c | 0
3 files changed, 1 insertion(+), 1 deletion(-)
rename {accel => target/i386}/mshv/msr.c (100%)accel/mshv/meson.build +0 -1
@@ -1,6 +1,5 @@ system_ss.add(when: 'CONFIG_MSHV', if_true: files( 'irq.c', 'mem.c', - 'msr.c', 'mshv-all.c' ))
target/i386/mshv/meson.build +1 -0
@@ -3,6 +3,7 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files( 'mshv-apic.c', 'mshv-cpu.c', + 'msr.c', )) i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
target/i386/mshv/msr.c renamed from accel/mshv/msr.c +0 -0
Signature · 1 line
2.34.1
PATCH 18/34 accel/mshv: store partition proc features
In reply to Magnus KulkeWe retrieve and store processor features on the state, so we can query them later when deciding which MSRs to migrate.
--- accel/mshv/mshv-all.c | 57 +++++++++++++++ include/hw/hyperv/hvhdk.h | 150 ++++++++++++++++++++++++++++++++++++++ include/system/mshv_int.h | 2 + 3 files changed, 209 insertions(+)
accel/mshv/mshv-all.c +57 -0
@@ -106,6 +106,57 @@ static int resume_vm(int vm_fd)return 0 ; } +static int get_partition_property (int vm_fd,uint32_t feature_bank, +uint64_t *value) +{ +struct hv_input_get_partition_property in = {0 }; +struct hv_output_get_partition_property out = {0 }; +struct mshv_root_hvcall args = {0 }; +int ret; + + in.property_code = feature_bank; + + args.code =HVCALL_GET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + args.out_sz =sizeof (out); + args.out_ptr = (uint64_t )&out; + + ret =ioctl (vm_fd,MSHV_ROOT_HVCALL , &args); +if (ret <0 ) { +error_report ("Failed to get guest partition property bank: %s" , +strerror (errno)); +return -1 ; + } + + *value = out.property_value; +return 0 ; +} + +static int get_proc_features (int vm_fd, +union hv_partition_processor_features *features) +{ +int ret; + + ret =get_partition_property (vm_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 , + features[0 ].as_uint64); +if (ret <0 ) { +error_report ("Failed to get processor features bank 0" ); +return -1 ; + } + + ret =get_partition_property (vm_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1 , + features[1 ].as_uint64); +if (ret <0 ) { +error_report ("Failed to get processor features bank 1" ); +return -1 ; + } + +return 0 ; +} +static int create_partition (int mshv_fd,int *vm_fd) {int ret;
@@ -441,6 +492,12 @@ static int mshv_init(AccelState *as, MachineState *ms) s->vm = vm_fd; s->fd = mshv_fd; + + ret =get_proc_features (vm_fd, &s->processor_features); +if (ret <0 ) { +return -1 ; + } + s->nr_as =1 ; s->as =g_new0 (MshvAddressSpace, s->nr_as);
include/hw/hyperv/hvhdk.h +150 -0
162 lines not on this page · raw message
include/system/mshv_int.h +2 -0
@@ -15,6 +15,7 @@#define QEMU_MSHV_INT_H #define MSHV_MSR_ENTRIES_COUNT 64 +#include "hw/hyperv/hvhdk.h" struct mshv_get_set_vp_state ;
@@ -55,6 +56,7 @@ struct MshvState {int nr_allocated_irq_routes;unsigned long *used_gsi_bitmap;unsigned int gsi_count; +union hv_partition_processor_features processor_features; }; typedef struct MshvMsiControl {
Signature · 1 line
2.34.1
PATCH 19/34 target/i386/mshv: expose msvh_get_generic_regs
In reply to Magnus KulkeWe expose the fn, so we can call them from the other source files (msr.c).
--- include/system/mshv_int.h | 2 ++ target/i386/mshv/mshv-cpu.c | 15 ++++++--------- 2 files changed, 8 insertions(+), 9 deletions(-)
include/system/mshv_int.h +2 -0
@@ -81,6 +81,8 @@ int mshv_configure_vcpu(const CPUState *cpu);int mshv_run_vcpu (int vm_fd,CPUState *cpu,hv_message *msg,MshvVmExit *exit);int mshv_set_generic_regs (const CPUState *cpu,const hv_register_assoc *assocs,size_t n_regs); +int mshv_get_generic_regs (CPUState *cpu,hv_register_assoc *assocs, +size_t n_regs);int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);
target/i386/mshv/mshv-cpu.c +6 -9
@@ -108,9 +108,6 @@ static enum hv_register_name FPU_REGISTER_NAMES[26] = { };static int set_special_regs (const CPUState *cpu); -static int get_generic_regs (CPUState *cpu, -struct hv_register_assoc *assocs, -size_t n_regs);static int get_lapic (CPUState *cpu) {
@@ -187,7 +184,7 @@ static int get_fpu(CPUState *cpu)for (size_t i =0 ; i < n_regs; i++) { assocs[i].name =FPU_REGISTER_NAMES [i]; } - ret =get_generic_regs (cpu, assocs, n_regs); + ret =mshv_get_generic_regs (cpu, assocs, n_regs);if (ret <0 ) {error_report ("failed to get special registers" );return -errno;
@@ -207,7 +204,7 @@ static int get_xc_reg(CPUState *cpu) assocs[0 ].name =HV_X64_REGISTER_XFEM ; - ret =get_generic_regs (cpu, assocs,1 ); + ret =mshv_get_generic_regs (cpu, assocs,1 );if (ret <0 ) {error_report ("failed to get xcr0" );return -1 ;
@@ -300,8 +297,8 @@ int mshv_set_generic_regs(const CPUState *cpu, const hv_register_assoc *assocs,return 0 ; } -static int get_generic_regs (CPUState *cpu,hv_register_assoc *assocs, -size_t n_regs) +int mshv_get_generic_regs (CPUState *cpu,hv_register_assoc *assocs, +size_t n_regs) {int cpu_fd =mshv_vcpufd (cpu);int vp_index = cpu->cpu_index;
@@ -450,7 +447,7 @@ static int get_standard_regs(CPUState *cpu)for (size_t i =0 ; i < n_regs; i++) { assocs[i].name =STANDARD_REGISTER_NAMES [i]; } - ret =get_generic_regs (cpu, assocs, n_regs); + ret =mshv_get_generic_regs (cpu, assocs, n_regs);if (ret <0 ) {error_report ("failed to get standard registers" );return -1 ;
@@ -527,7 +524,7 @@ static int get_special_regs(CPUState *cpu)for (size_t i =0 ; i < n_regs; i++) { assocs[i].name =SPECIAL_REGISTER_NAMES [i]; } - ret =get_generic_regs (cpu, assocs, n_regs); + ret =mshv_get_generic_regs (cpu, assocs, n_regs);if (ret <0 ) {error_report ("failed to get special registers" );return -errno;
Signature · 1 line
2.34.1
PATCH 20/34 target/i386/mshv: migrate MSRs
In reply to Magnus KulkeIn this change the we rewrite the existing MSR logic to make MSRs migratable:
- we map them on existing QEMU fields in the CPU. A table and a macro MSHV_ENV_FIELD is used to associate a HV register name to the their msr index and their offset in the cpu state struct. The list is not exhaustive and will be extended in follow-up commits.
- mshv_set/get_msrs() fns are called in the arch_load/store_vcpu_state() fns. they use use generic registers ioctl's and map the input/output via load/store_to/from_env() from/to the hv register content to the cpu state representation.
- init_msrs() has been moved from mshv-vcpu to the msr source file
- we need to perform some filtering of MSR because before writing and reading, because the hvcalls will fail if the partition doesn't support a given MSRs.
- Some MSRs are partition-wide and so we will only write the to on the BSP.
--- include/hw/hyperv/hvgdk_mini.h | 16 + include/system/mshv_int.h | 17 +- target/i386/mshv/mshv-cpu.c | 40 +-- target/i386/mshv/msr.c | 554 ++++++++++++++------------------- 4 files changed, 259 insertions(+), 368 deletions(-)
include/hw/hyperv/hvgdk_mini.h +16 -0
@@ -9,6 +9,19 @@#define MSHV_IOCTL 0xB8 +/* Hyper-V specific model specific registers (MSRs) */ + +/* HV_X64_SYNTHETIC_MSR */ +#define HV_X64_MSR_GUEST_OS_ID 0x40000000 +#define HV_X64_MSR_HYPERCALL 0x40000001 +#define HV_X64_MSR_VP_INDEX 0x40000002 +#define HV_X64_MSR_RESET 0x40000003 +#define HV_X64_MSR_VP_RUNTIME 0x40000010 +#define HV_X64_MSR_TIME_REF_COUNT 0x40000020 +#define HV_X64_MSR_REFERENCE_TSC 0x40000021 +#define HV_X64_MSR_TSC_FREQUENCY 0x40000022 +#define HV_X64_MSR_APIC_FREQUENCY 0x40000023 + typedef enum hv_register_name { /* Pending Interruption Register */ HV_REGISTER_PENDING_INTERRUPTION = 0x00010002,
@@ -152,12 +165,14 @@ typedef enum hv_register_name {/* Available */ HV_X64_REGISTER_SPEC_CTRL =0x00080084 , +HV_X64_REGISTER_TSC_DEADLINE =0x00080095 ,HV_X64_REGISTER_TSC_ADJUST =0x00080096 ,/* Other MSRs */ HV_X64_REGISTER_MSR_IA32_MISC_ENABLE =0x000800A0 ,/* Misc */ +HV_X64_REGISTER_HYPERCALL =0x00090001 ,HV_REGISTER_GUEST_OS_ID =0x00090002 , HV_REGISTER_REFERENCE_TSC = 0x00090017,
@@ -797,6 +812,7 @@ struct hv_cpuid {#define IA32_MSR_DEBUG_CTL 0x1D9#define IA32_MSR_SPEC_CTRL 0x00000048#define IA32_MSR_TSC_ADJUST 0x0000003b +#define IA32_MSR_TSC_DEADLINE 0x000006e0#define IA32_MSR_MISC_ENABLE 0x000001a0
include/system/mshv_int.h +3 -14
@@ -14,7 +14,6 @@#ifndef QEMU_MSHV_INT_H #define QEMU_MSHV_INT_H -#define MSHV_MSR_ENTRIES_COUNT 64#include "hw/hyperv/hvhdk.h" struct mshv_get_set_vp_state ;
@@ -116,18 +115,8 @@ void mshv_set_phys_mem(MshvMemoryListener *mml, MemoryRegionSection *section, bool add);/* msr */ -typedef struct MshvMsrEntry { -uint32_t index; -uint32_t reserved; -uint64_t data; -}MshvMsrEntry ; - -typedef struct MshvMsrEntries { -MshvMsrEntry entries[MSHV_MSR_ENTRIES_COUNT ]; -uint32_t nmsrs; -}MshvMsrEntries ; - -int mshv_configure_msr (const CPUState *cpu,const MshvMsrEntry *msrs, -size_t n_msrs); +int mshv_init_msrs (const CPUState *cpu); +int mshv_get_msrs (CPUState *cpu); +int mshv_set_msrs (const CPUState *cpu);#endif
target/i386/mshv/mshv-cpu.c +12 -28
@@ -580,6 +580,11 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } + ret =mshv_get_msrs (cpu); +if (ret <0 ) { +return ret; + } +return 0 ; }
@@ -1025,6 +1030,12 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } +/* INVARIANT: LAPIC must be restored before MSRs (TSC_DEADLINE) */ + ret =mshv_set_msrs (cpu); +if (ret <0 ) { +return ret; + } +return 0 ; }
@@ -1511,33 +1522,6 @@ void mshv_init_mmio_emu(void)init_emu (&mshv_x86_emul_ops); } -static int init_msrs (const CPUState *cpu) -{ -int ret; -uint64_t d_t =MSR_MTRR_ENABLE |MSR_MTRR_MEM_TYPE_WB ; - -const struct hv_register_assoc assocs[] = { - { .name =HV_X64_REGISTER_SYSENTER_CS , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_SYSENTER_ESP , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_SYSENTER_EIP , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_STAR , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_CSTAR , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_LSTAR , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_KERNEL_GS_BASE , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_SFMASK , .value.reg64 =0x0 }, - { .name =HV_X64_REGISTER_MSR_MTRR_DEF_TYPE , .value.reg64 = d_t }, - }; -QEMU_BUILD_BUG_ON (ARRAY_SIZE (assocs) >MSHV_MSR_ENTRIES_COUNT ); - - ret =mshv_set_generic_regs (cpu, assocs,ARRAY_SIZE (assocs)); -if (ret <0 ) { -error_report ("failed to put msrs" ); -return -1 ; - } - -return 0 ; -} -void mshv_arch_init_vcpu (CPUState *cpu) {X86CPU *x86_cpu =X86_CPU (cpu);
@@ -1567,7 +1551,7 @@ void mshv_arch_init_vcpu(CPUState *cpu) ret =init_cpuid2 (cpu);assert (ret ==0 ); - ret =init_msrs (cpu); + ret =mshv_init_msrs (cpu);assert (ret ==0 ); ret =init_lint (cpu);
target/i386/mshv/msr.c +228 -326
590 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 21/34 target/i386/mshv: migrate MTRR MSRs
In reply to Magnus KulkeThis change roundtrips memory access/caching MSRs. The mapping scheme is a bit more elaborate on these, so we have added a special handling instead of individual entries in the MSR mapping table.
--- target/i386/mshv/msr.c | 136 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 129 insertions(+), 7 deletions(-)
target/i386/mshv/msr.c +129 -7
@@ -79,6 +79,10 @@ static const MshvMsrEnvMap msr_env_map[] = { {HV_X64_MSR_SIMP ,HV_REGISTER_SIMP , offsetof(CPUX86State, msr_hv_synic_msg_page) }, +/* MTRR default type */ + { IA32_MSR_MTRR_DEF_TYPE, HV_X64_REGISTER_MSR_MTRR_DEF_TYPE, + offsetof(CPUX86State, mtrr_deftype) }, +/* Other */ /* TODO: find out processor features that correlate to unsupported MSRs. */
@@ -90,6 +94,98 @@ static const MshvMsrEnvMap msr_env_map[] = { offsetof(CPUX86State, spec_ctrl) }, }; +/* +* The assocs have to be set according to this schema: +* 8 entries for 0-7 mtrr_base +* 8 entries for mtrr_mask 0-7 +* 11 entries for 1 x 64k, 2 x 16k, 8 x 4k fixed MTRR +* 27 total entries +*/ + +#define MSHV_MTRR_MSR_COUNT 27 +#define MSHV_MSR_TOTAL_COUNT (ARRAY_SIZE(msr_env_map) + MSHV_MTRR_MSR_COUNT) + +static void store_in_env_mtrr_phys (CPUState *cpu, +const struct hv_register_assoc *assocs, +size_t n_assocs) +{ +X86CPU *x86_cpu =X86_CPU (cpu); +CPUX86State *env = &x86_cpu->env; +size_t i, fixed_offset; +hv_register_name hv_name; +uint64_t base, mask; + +assert (n_assocs ==MSHV_MTRR_MSR_COUNT ); + +for (i =0 ; i < MSR_MTRRcap_VCNT; i++) { + hv_name =HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i; +assert (assocs[i].name == hv_name); + hv_name =HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i; +assert (assocs[i + MSR_MTRRcap_VCNT].name == hv_name); + + base = assocs[i].value.reg64; + mask = assocs[i + MSR_MTRRcap_VCNT].value.reg64; + env->mtrr_var[i].base = base; + env->mtrr_var[i].mask = mask; + } + +/* fixed 1x 64, 2x 16, 8x 4 kB */ + fixed_offset = MSR_MTRRcap_VCNT *2 ; +for (i =0 ; i <11 ; i++) { + hv_name =HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i; +assert (assocs[fixed_offset + i].name == hv_name); + env->mtrr_fixed[i] = assocs[fixed_offset + i].value.reg64; + } +} + +/* +* The assocs have to be set according to this schema: +* 8 entries for 0-7 mtrr_base +* 8 entries for mtrr_mask 0-7 +* 11 entries for 1 x 64k, 2 x 16k, 8 x 4k fixed MTRR +* 27 total entries +*/ +static void load_from_env_mtrr_phys (const CPUState *cpu, +struct hv_register_assoc *assocs, +size_t n_assocs) +{ +X86CPU *x86_cpu =X86_CPU (cpu); +CPUX86State *env = &x86_cpu->env; +size_t i, fixed_offset; +uint64_t base, mask, fixed_value; +hv_register_name base_name, mask_name, fixed_name; +hv_register_assoc *assoc; + +assert (n_assocs ==MSHV_MTRR_MSR_COUNT ); + +for (i =0 ; i < MSR_MTRRcap_VCNT; i++) { + base = env->mtrr_var[i].base; + mask = env->mtrr_var[i].mask; + + base_name =HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i; + mask_name =HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i; + + assoc = &assocs[i]; + assoc->name = base_name; + assoc->value.reg64 = base; + + assoc = &assocs[i + MSR_MTRRcap_VCNT]; + assoc->name = mask_name; + assoc->value.reg64 = mask; + } + +/* fixed 1x 64, 2x 16, 8x 4 kB */ + fixed_offset = MSR_MTRRcap_VCNT *2 ; +for (i =0 ; i <11 ; i++) { + fixed_name =HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i; + fixed_value = env->mtrr_fixed[i]; + + assoc = &assocs[fixed_offset + i]; + assoc->name = fixed_name; + assoc->value.reg64 = fixed_value; + } +} +int mshv_init_msrs (const CPUState *cpu) {int ret;
@@ -131,8 +227,9 @@ static void store_in_env(CPUState *cpu, const struct hv_register_assoc *assocs,union hv_register_value hv_value;ptrdiff_t offset;uint32_t hv_name; +size_t mtrr_index; -assert (n_assocs <= (ARRAY_SIZE (msr_env_map))); +assert (n_assocs <=MSHV_MSR_TOTAL_COUNT );for (i =0 , j =0 ; i <ARRAY_SIZE (msr_env_map); i++) { hv_name = assocs[j].name;
@@ -146,17 +243,38 @@ static void store_in_env(CPUState *cpu, const struct hv_register_assoc *assocs,MSHV_ENV_FIELD (env, offset) = hv_value.reg64; j++; } + + mtrr_index = j; +store_in_env_mtrr_phys (cpu, &assocs[mtrr_index],MSHV_MTRR_MSR_COUNT ); }static void set_hv_name_in_assocs (struct hv_register_assoc *assocs,size_t n_assocs) {size_t i; +size_t mtrr_offset, mtrr_fixed_offset; +hv_register_name hv_name; + +assert (n_assocs ==MSHV_MSR_TOTAL_COUNT ); -assert (n_assocs ==ARRAY_SIZE (msr_env_map));for (i =0 ; i <ARRAY_SIZE (msr_env_map); i++) { assocs[i].name = msr_env_map[i].hv_name; } + + mtrr_offset =ARRAY_SIZE (msr_env_map); +for (i =0 ; i < MSR_MTRRcap_VCNT; i++) { + hv_name =HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i; + assocs[mtrr_offset + i].name = hv_name; + hv_name =HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i; + assocs[mtrr_offset + MSR_MTRRcap_VCNT + i].name = hv_name; + } + +/* fixed 1x 64, 2x 16, 8x 4 kB */ + mtrr_fixed_offset = mtrr_offset + MSR_MTRRcap_VCNT *2 ; +for (i =0 ; i <11 ; i++) { + hv_name =HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i; + assocs[mtrr_fixed_offset + i].name = hv_name; + } }static bool msr_supported (uint32_t name)
@@ -181,8 +299,8 @@ static bool msr_supported(uint32_t name)int mshv_get_msrs (CPUState *cpu) {int ret =0 ; -size_t n_assocs =ARRAY_SIZE (msr_env_map); -struct hv_register_assoc assocs[ARRAY_SIZE (msr_env_map)]; +size_t n_assocs =MSHV_MSR_TOTAL_COUNT ; +struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];size_t i, j;uint32_t name;
@@ -223,8 +341,9 @@ static void load_from_env(const CPUState *cpu, struct hv_register_assoc *assocs,CPUX86State *env = &x86_cpu->env;ptrdiff_t offset;union hv_register_value *hv_value; +size_t mtrr_offset; -assert (n_assocs ==ARRAY_SIZE (msr_env_map)); +assert (n_assocs ==MSHV_MSR_TOTAL_COUNT );for (i =0 ; i <ARRAY_SIZE (msr_env_map); i++) { mapping = &msr_env_map[i];
@@ -233,12 +352,15 @@ static void load_from_env(const CPUState *cpu, struct hv_register_assoc *assocs, hv_value = &assocs[i].value; hv_value->reg64 =MSHV_ENV_FIELD (env, offset); } + + mtrr_offset =ARRAY_SIZE (msr_env_map); +load_from_env_mtrr_phys (cpu, &assocs[mtrr_offset],MSHV_MTRR_MSR_COUNT ); }int mshv_set_msrs (const CPUState *cpu) { -size_t n_assocs =ARRAY_SIZE (msr_env_map); -struct hv_register_assoc assocs[ARRAY_SIZE (msr_env_map)]; +size_t n_assocs =MSHV_MSR_TOTAL_COUNT ; +struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];int ret;size_t i, j;
Signature · 1 line
2.34.1
PATCH 22/34 target/i386/mshv: migrate Synic SINT MSRs
In reply to Magnus KulkeMigrate HyperV SynIC SINT MSRs. We can only read/write those if SCONTROL is enabled in the guest, hence we have to split the SINT MSR out and make reading/writing them dependent on that MSR.
--- target/i386/mshv/msr.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
target/i386/mshv/msr.c +41 -0
@@ -303,6 +303,8 @@ int mshv_get_msrs(CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];size_t i, j;uint32_t name; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);
@@ -329,6 +331,27 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs); +/* Read SINT MSRs only if SynIC is enabled */ + synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + } + + ret =mshv_get_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to get SynIC SINT MSRs" ); +return -errno; + } + +for (i =0 ; i <HV_SINT_COUNT ; i++) { +uint64_t hv_sint_value = assocs[i].value.reg64; + x86cpu->env.msr_hv_synic_sint[i] = hv_sint_value; + } + } +return 0 ; }
@@ -363,6 +386,8 @@ int mshv_set_msrs(const CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];int ret;size_t i, j; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);
@@ -395,5 +420,21 @@ int mshv_set_msrs(const CPUState *cpu)return -errno; } +/* SINT MSRs can only be written if SCONTROL has been set, so we split */ +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + assocs[i].value.reg64 = x86cpu->env.msr_hv_synic_sint[i]; + } + + ret =mshv_set_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to set SynIC SINT MSRs" ); +return -errno; + } + } +return 0 ; }
Signature · 1 line
2.34.1
PATCH 23/34 target/i386/mshv: migrate CET/SS MSRs
In reply to Magnus KulkeThis change migrates the MSRs required for CET shadow stack and indirect branch tracking. They are gated behind cet_ss_support || cet_ibt_support mshv processor feature flags.
--- include/hw/hyperv/hvgdk_mini.h | 11 +++++++++++ target/i386/mshv/msr.c | 30 ++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+)
include/hw/hyperv/hvgdk_mini.h +11 -0
@@ -168,6 +168,17 @@ typedef enum hv_register_name {HV_X64_REGISTER_TSC_DEADLINE =0x00080095 ,HV_X64_REGISTER_TSC_ADJUST =0x00080096 , +/* CET / Shadow Stack */ +HV_X64_REGISTER_U_XSS =0x0008008B , +HV_X64_REGISTER_U_CET =0x0008008C , +HV_X64_REGISTER_S_CET =0x0008008D , +HV_X64_REGISTER_SSP =0x0008008E , +HV_X64_REGISTER_PL0_SSP =0x0008008F , +HV_X64_REGISTER_PL1_SSP =0x00080090 , +HV_X64_REGISTER_PL2_SSP =0x00080091 , +HV_X64_REGISTER_PL3_SSP =0x00080092 , +HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR =0x00080093 , +/* Other MSRs */ HV_X64_REGISTER_MSR_IA32_MISC_ENABLE = 0x000800A0,
target/i386/mshv/msr.c +30 -0
@@ -83,6 +83,26 @@ static const MshvMsrEnvMap msr_env_map[] = { {IA32_MSR_MTRR_DEF_TYPE ,HV_X64_REGISTER_MSR_MTRR_DEF_TYPE , offsetof(CPUX86State, mtrr_deftype) }, +/* CET / Shadow Stack */ + { MSR_IA32_U_CET, HV_X64_REGISTER_U_CET, + offsetof(CPUX86State, u_cet) }, + { MSR_IA32_S_CET, HV_X64_REGISTER_S_CET, + offsetof(CPUX86State, s_cet) }, + { MSR_IA32_PL0_SSP, HV_X64_REGISTER_PL0_SSP, + offsetof(CPUX86State, pl0_ssp) }, + { MSR_IA32_PL1_SSP, HV_X64_REGISTER_PL1_SSP, + offsetof(CPUX86State, pl1_ssp) }, + { MSR_IA32_PL2_SSP, HV_X64_REGISTER_PL2_SSP, + offsetof(CPUX86State, pl2_ssp) }, + { MSR_IA32_PL3_SSP, HV_X64_REGISTER_PL3_SSP, + offsetof(CPUX86State, pl3_ssp) }, + { MSR_IA32_INT_SSP_TAB, HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR, + offsetof(CPUX86State, int_ssp_table) }, + +/* XSAVE Supervisor State */ + { MSR_IA32_XSS, HV_X64_REGISTER_U_XSS, + offsetof(CPUX86State, xss) }, +/* Other */ /* TODO: find out processor features that correlate to unsupported MSRs. */
@@ -291,6 +311,16 @@ static bool msr_supported(uint32_t name)return mshv_state->processor_features.tsc_adjust_support;case HV_X64_REGISTER_TSC_DEADLINE :return mshv_state->processor_features.tsc_deadline_tmr_support; +case HV_X64_REGISTER_U_CET : +case HV_X64_REGISTER_S_CET : +case HV_X64_REGISTER_PL0_SSP : +case HV_X64_REGISTER_PL1_SSP : +case HV_X64_REGISTER_PL2_SSP : +case HV_X64_REGISTER_PL3_SSP : +case HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR : +case HV_X64_REGISTER_U_XSS : +return mshv_state->processor_features.cet_ss_support || + mshv_state->processor_features.cet_ibt_support; }return true;
Signature · 1 line
2.34.1
PATCH 24/34 target/i386/mshv: migrate SIMP and SIEFP state
In reply to Magnus KulkeThis part SynIC state is retrieved from the hypervisor via aligned state pages:
- Add new synic source file
- Centralize the synic_enabled() check
- r/w pages from the hyper via aligned pages
- only handle pages when synic is enabled
- add buffers for migration to VM state
--- include/system/mshv_int.h | 7 ++ target/i386/cpu.h | 5 ++ target/i386/machine.c | 26 ++++++ target/i386/mshv/meson.build | 1 + target/i386/mshv/mshv-cpu.c | 64 +++++++++++++++ target/i386/mshv/msr.c | 7 +- target/i386/mshv/synic.c | 155 +++++++++++++++++++++++++++++++++++ 7 files changed, 260 insertions(+), 5 deletions(-) create mode 100644 target/i386/mshv/synic.c
include/system/mshv_int.h +7 -0
@@ -119,4 +119,11 @@ int mshv_init_msrs(const CPUState *cpu);int mshv_get_msrs (CPUState *cpu);int mshv_set_msrs (const CPUState *cpu); +/* synic */ +int mshv_get_simp (int cpu_fd,uint8_t *page); +int mshv_set_simp (int cpu_fd,const uint8_t *page); +int mshv_get_siefp (int cpu_fd,uint8_t *page); +int mshv_set_siefp (int cpu_fd,const uint8_t *page); +bool mshv_synic_enabled (const CPUState *cpu); +#endif
target/i386/cpu.h +5 -0
@@ -33,6 +33,7 @@#include "qemu/cpu-float.h" #include "qemu/timer.h" #include "standard-headers/asm-x86/kvm_para.h" +#include "hw/hyperv/hvgdk_mini.h" #define XEN_NR_VIRQS 24
@@ -2291,6 +2292,10 @@ typedef struct CPUArchState {#if defined(CONFIG_HVF ) || defined(CONFIG_MSHV ) || defined(CONFIG_WHPX )void *emu_mmio_buf;#endif +#if defined(CONFIG_MSHV ) +uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +#endif uint64_t mcg_cap;uint64_t mcg_ctl;
target/i386/machine.c +26 -0
@@ -952,6 +952,29 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { } }; +#ifdef CONFIG_MSHV +static bool mshv_synic_vp_state_needed (void *opaque) +{ +X86CPU *cpu = opaque; +CPUX86State *env = &cpu->env; + +/* Only migrate SIMP/SIEFP if SynIC is enabled */ +return env->msr_hv_synic_control &1 ; +} + +static const VMStateDescription vmstate_mshv_synic_vp_state = { + .name ="cpu/mshv_synic_vp_state" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synic_vp_state_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_simp_page,X86CPU ), +VMSTATE_BUFFER (env.hv_siefp_page,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +#endif +static bool avx512_needed (void *opaque) {X86CPU *cpu = opaque;
@@ -1916,6 +1939,9 @@ const VMStateDescription vmstate_x86_cpu = { &vmstate_cet,#ifdef TARGET_X86_64 &vmstate_apx, +#endif +#ifdef CONFIG_MSHV + &vmstate_mshv_synic_vp_state, #endifNULL }
target/i386/mshv/meson.build +1 -0
@@ -4,6 +4,7 @@ i386_mshv_ss.add(files( 'mshv-apic.c', 'mshv-cpu.c', 'msr.c', + 'synic.c', )) i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
target/i386/mshv/mshv-cpu.c +64 -0
88 lines not on this page · raw message
target/i386/mshv/msr.c +2 -5
@@ -334,7 +334,6 @@ int mshv_get_msrs(CPUState *cpu)size_t i, j;uint32_t name;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);
@@ -362,8 +361,7 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs);/* Read SINT MSRs only if SynIC is enabled */ - synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {
@@ -417,7 +415,6 @@ int mshv_set_msrs(const CPUState *cpu)int ret;size_t i, j;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);
@@ -451,7 +448,7 @@ int mshv_set_msrs(const CPUState *cpu) }/* SINT MSRs can only be written if SCONTROL has been set, so we split */ -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {
target/i386/mshv/synic.c new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 25/34 target/i386/mshv: migrate STIMER state
In reply to Magnus KulkeThis part of Synic state is retrieved via a mem-aligned page. We declare the required space (size reference: rust-vmm/mshv) as a buffer on the VM state struct for inclusion in a migration.
Other than other SynIC features, STIMER doesn't depend on SCONTROL being set.
--- include/system/mshv_int.h | 2 ++ target/i386/cpu.h | 5 ++++ target/i386/machine.c | 20 +++++++++++++++ target/i386/mshv/mshv-cpu.c | 12 +++++++++ target/i386/mshv/synic.c | 51 +++++++++++++++++++++++++++++++++++++ 5 files changed, 90 insertions(+)
include/system/mshv_int.h +2 -0
@@ -125,5 +125,7 @@ int mshv_set_simp(int cpu_fd, const uint8_t *page);int mshv_get_siefp (int cpu_fd,uint8_t *page);int mshv_set_siefp (int cpu_fd,const uint8_t *page);bool mshv_synic_enabled (const CPUState *cpu); +int mshv_get_synthetic_timers (int cpu_fd,uint8_t *state); +int mshv_set_synthetic_timers (int cpu_fd,const uint8_t *state);#endif
target/i386/cpu.h +5 -0
@@ -45,6 +45,10 @@#define ELF_MACHINE_UNAME "i686"#endif +#ifdef CONFIG_MSHV +#define MSHV_STIMERS_STATE_SIZE 200 +#endif +enum {R_EAX =0 ,R_ECX =1 ,
@@ -2295,6 +2299,7 @@ typedef struct CPUArchState {#if defined(CONFIG_MSHV )uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ];uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_synthetic_timers_state[MSHV_STIMERS_STATE_SIZE ];#endif uint64_t mcg_cap;
target/i386/machine.c +20 -0
@@ -10,6 +10,7 @@#include "exec/watchpoint.h" #include "system/kvm.h" #include "system/kvm_xen.h" +#include "system/mshv.h" #include "system/tcg.h" #include "qemu/error-report.h"
@@ -953,6 +954,24 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { };#ifdef CONFIG_MSHV + +static bool mshv_synthetic_timers_needed (void *opaque) +{ +/* Always migrate synthetic timers */ +return mshv_enabled (); +} + +static const VMStateDescription vmstate_mshv_synthetic_timers = { + .name ="cpu/mshv_synthetic_timers" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synthetic_timers_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_synthetic_timers_state,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +static bool mshv_synic_vp_state_needed (void *opaque) {X86CPU *cpu = opaque;
@@ -1942,6 +1961,7 @@ const VMStateDescription vmstate_x86_cpu = {#endif #ifdef CONFIG_MSHV &vmstate_mshv_synic_vp_state, + &vmstate_mshv_synthetic_timers, #endifNULL }
target/i386/mshv/mshv-cpu.c +12 -0
@@ -135,6 +135,12 @@ static int get_synic_state(CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_get_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to get synthetic timers" ); +return -1 ; + } +/* SIMP/SIEFP can only be read when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;
@@ -1039,6 +1045,12 @@ static int set_synic_state(const CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_set_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to set synthetic timers state" ); +return -1 ; + } +/* SIMP/SIEFP can only be written when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;
target/i386/mshv/synic.c +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 26/34 accel/mshv: introduce SaveVMHandler
In reply to Magnus KulkeThis mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.
Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.
--- accel/mshv/mshv-all.c | 75 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)
accel/mshv/mshv-all.c +75 -0
@@ -39,6 +39,8 @@#include "system/mshv.h" #include "system/mshv_int.h" #include "system/reset.h" +#include "migration/qemu-file-types.h" +#include "migration/register.h" #include "trace.h" #include <err.h> #include <sys/ioctl.h>
@@ -47,6 +49,9 @@ bool mshv_allowed;MshvState *mshv_state; +static int pause_vm (int vm_fd); +static int resume_vm (int vm_fd); +static int init_mshv (int *mshv_fd) {int fd =open ("/dev/mshv" ,O_RDWR |O_CLOEXEC );
@@ -80,6 +85,64 @@ static int set_time_freeze(int vm_fd, int freeze)return 0 ; } +static void mshv_save_state (QEMUFile *f,void *opaque) +{ +return ; +} + +static int mshv_load_state (QEMUFile *f,void *opaque,int version_id) +{ +return 0 ; +} + +static int mshv_save_prepare (void *opaque,Error **errp) +{ +MshvState *s = opaque; +int ret; + + ret =pause_vm (s->vm); +if (ret <0 ) { +error_setg (errp,"Failed to pause VM for migration" ); +return -1 ; + } + +return 0 ; +} + +static void mshv_save_cleanup (void *opaque) +{ +MshvState *s = opaque; +int ret; + + ret =resume_vm (s->vm); +if (ret <0 ) { +error_report ("Failed to resme VM after migration" ); + } +} + +static int mshv_load_setup (QEMUFile *f,void *opaque,Error **errp) +{ +MshvState *s = opaque; +int ret; + + ret =pause_vm (s->vm); +if (ret <0 ) { +error_setg (errp,"Failed to pause VM for migration restore" ); +return -1 ; + } + +return 0 ; +} + +static int mshv_load_cleanup (void *opaque) +{ +MshvState *s = opaque; + +resume_vm (s->vm); + +return 0 ; +} +static int pause_vm (int vm_fd) {int ret;
@@ -454,6 +517,15 @@ static int mshv_init_vcpu(CPUState *cpu)return 0 ; } +static SaveVMHandlers savevm_mshv = { + .save_prepare = mshv_save_prepare, + .save_state = mshv_save_state, + .save_cleanup = mshv_save_cleanup, + .load_setup = mshv_load_setup, + .load_state = mshv_load_state, + .load_cleanup = mshv_load_cleanup, +}; +static int mshv_init (AccelState *as,MachineState *ms) {MshvState *s;
@@ -509,6 +581,9 @@ static int mshv_init(AccelState *as, MachineState *ms) 0, "mshv-memory");memory_listener_register (&mshv_io_listener, &address_space_io); +/* register custom handlers for migration events */ +register_savevm_live ("mshv" ,0 ,1 , &savevm_mshv, s); +return 0 ; }
Signature · 1 line
2.34.1
Paolo Bonzini
maintainerRe: [PATCH 26/34] accel/mshv: introduce SaveVMHandler
Il ven 17 apr 2026, 12:58 Magnus Kulke <[email protected]> ha scritto:
Quoted 7 lines
This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.
Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.
Sorry, I forgot to reply about this one.
It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.
Paolo
Quoted 122 lines
--- accel/mshv/mshv-all.c | 75 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+)accel/mshv/mshv-all.c +75
-1@@ -39,6 +39,8 @@#include "system/mshv.h" #include "system/mshv_int.h" #include "system/reset.h" +#include "migration/qemu-file-types.h" +#include "migration/register.h" #include "trace.h" #include <err.h> #include <sys/ioctl.h> @@ -47,6 +49,9 @@ bool mshv_allowed;MshvState *mshv_state; +static int pause_vm (int vm_fd); +static int resume_vm (int vm_fd); +static int init_mshv (int *mshv_fd) {int fd =open ("/dev/mshv" ,O_RDWR |O_CLOEXEC );@@ -80,6 +85,64 @@ static int set_time_freeze(int vm_fd, int freeze)return 0 ; } +static void mshv_save_state (QEMUFile *f,void *opaque) +{ +return ; +} + +static int mshv_load_state (QEMUFile *f,void *opaque,int version_id) +{ +return 0 ; +} + +static int mshv_save_prepare (void *opaque,Error **errp) +{ +MshvState *s = opaque; +int ret; + + ret =pause_vm (s->vm); +if (ret <0 ) { +error_setg (errp,"Failed to pause VM for migration" ); +return -1 ; + } + +return 0 ; +} + +static void mshv_save_cleanup (void *opaque) +{ +MshvState *s = opaque; +int ret; + + ret =resume_vm (s->vm); +if (ret <0 ) { +error_report ("Failed to resme VM after migration" ); + } +} + +static int mshv_load_setup (QEMUFile *f,void *opaque,Error **errp) +{ +MshvState *s = opaque; +int ret; + + ret =pause_vm (s->vm); +if (ret <0 ) { +error_setg (errp,"Failed to pause VM for migration restore" ); +return -1 ; + } + +return 0 ; +} + +static int mshv_load_cleanup (void *opaque) +{ +MshvState *s = opaque; + +resume_vm (s->vm); + +return 0 ; +} +static int pause_vm (int vm_fd) {int ret;@@ -454,6 +517,15 @@ static int mshv_init_vcpu(CPUState *cpu)return 0 ; } +static SaveVMHandlers savevm_mshv = { + .save_prepare = mshv_save_prepare, + .save_state = mshv_save_state, + .save_cleanup = mshv_save_cleanup, + .load_setup = mshv_load_setup, + .load_state = mshv_load_state, + .load_cleanup = mshv_load_cleanup, +}; +static int mshv_init (AccelState *as,MachineState *ms) {MshvState *s;@@ -509,6 +581,9 @@ static int mshv_init(AccelState *as, MachineState *ms) 0, "mshv-memory");memory_listener_register (&mshv_io_listener, &address_space_io); +/* register custom handlers for migration events */ +register_savevm_live ("mshv" ,0 ,1 , &savevm_mshv, s); +return 0 ; } --2.34.1
Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler
Quoted 11 lines · Paolo Bonzini
Il ven 17 apr 2026, 12:58 Magnus Kulke <[email protected]> ha scritto:
Quoted 7 lines
This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.
Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.
Quoted 8 lines
Sorry, I forgot to reply about this one.
It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.
Oh, good catch, thanks. I misunderstood something about those hooks apparently. I think this was originally inspired by migration logic in Cloud-Hypervisor, which freezes/unfreezes a partition during migration
- but only after the pre-copy phase is done.
I assumed the hooks were also called only before the final delta is transmitted. In tests I was relying on the reported state of the source from QMP, which probably can not accurately reflect the state of the partition as the freeze/unfreeze happens out of band (i.e. is is not orchestrated by the migration framework).
I'll have to research a bit and see if there are more appropriate hooks, or if we don't need to freeze the source partition at all.
best,
magnus
Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler
In reply to Paolo BonziniQuoted 10 lines · Paolo Bonzini
Sorry, I forgot to reply about this one.
It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.
Paolo
Hey Paolo,
I spent some time looking at this. it's a bit twisted. The un/pause() calls are unfortunately named, they are merely freezing the progress of the reference timer, which for any real-world workload is in fact pausing the guest b/c if the timer doesn't progress an os would mostly stall, waiting for ticks.
We want to freeze the timer though during the last phase of the migration on the source and on the destination (can't set the reference time if the vcpu time is not frozen). But currently it's the wrong place. I'll find some more fitting place to perform this ceremony.
Curiously the time-frozen source was obscuring another potentially more serious bug. The mshv vcpu dispatch loop currently does not honor a cpu->exit_request, like it is currently set after precopy:
migration_completion_precopy()
migration_stop_vm()
vm_stop_force_state()
vm_stop()
do_vm_stop()
pause_all_vcpus()
cpu_pause()
cpu_exit()
qatomic_store_release(&cpu->exit_request, true)The two bugs were sort of cancelling each other out. Since the source guest was not busy, migrations were still succeeding, even though the the vcpus were not actually "stopped" on the vcpu.
I'll rework this a bit, and submit a revised version.
best,
magnus
Paolo Bonzini
maintainerRe: [PATCH 26/34] accel/mshv: introduce SaveVMHandler
Il ven 3 lug 2026, 19:22 Magnus Kulke <[email protected]> ha scritto:
Quoted 13 lines · Paolo Bonzini
Quoted 5 lines
Sorry, I forgot to reply about this one.
It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while
the
Quoted 5 lines
VM runs, and the SaveVMHandler should just fail migration of the VM is running.
Paolo
Quoted 8 lines
Hey Paolo,
I spent some time looking at this. it's a bit twisted. The un/pause() calls are unfortunately named, they are merely freezing the progress of the reference timer, which for any real-world workload is in fact pausing the guest b/c if the timer doesn't progress an os would mostly stall, waiting for ticks.
There are different "schools of thought" for that. Freezing introduces an incorrect timing on the destination, not freezing introduces a jump.
If you choose to freeze that's fine, QEMU does the same for KVM even though KVM has alternative ioctls that allow not freezing.
But currently it's the wrong
Quoted 2 lines
place. I'll find some more fitting place to perform this ceremony.
KVM uses a device for that, see kvnclock.c.
Curiously the time-frozen source was obscuring another potentially more
Quoted 2 lines
serious bug. The mshv vcpu dispatch loop currently does not honor a
cpu->exit_request
Was that related to the other patch that was touching the vcpu thread loop?
Paolo
like it is currently set after precopy:
Quoted 22 lines
migration_completion_precopy()
migration_stop_vm()vm_stop_force_state() vm_stop() do_vm_stop() pause_all_vcpus() cpu_pause() cpu_exit() qatomic_store_release(&cpu->exit_request, true)The two bugs were sort of cancelling each other out. Since the source guest was not busy, migrations were still succeeding, even though the the vcpus were not actually "stopped" on the vcpu.
I'll rework this a bit, and submit a revised version.
best,
magnus
Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler
Quoted 11 lines · Paolo Bonzini
There are different "schools of thought" for that. Freezing introduces an incorrect timing on the destination, not freezing introduces a jump.
If you choose to freeze that's fine, QEMU does the same for KVM even though KVM has alternative ioctls that allow not freezing.
But currently it's the wrong
Quoted 2 lines
place. I'll find some more fitting place to perform this ceremony.
KVM uses a device for that, see
kvnclock.c.
Makes sense, kvmclock looks like a good template. I'll use a similar object.
Quoted 8 lines
Curiously the time-frozen source was obscuring another potentially more
Quoted 2 lines
serious bug. The mshv vcpu dispatch loop currently does not honor a
cpu->exit_requestWas that related to the other patch that was touching the vcpu thread loop?
yup, we need the same here, I'll address it as part of a migration-pt2 series. we only need to check for cpu->exit_request and break the loop if set.
best,
magnus
PATCH 27/34 accel/mshv: write synthetic MSRs after migration
In reply to Magnus KulkeWrite partition-wide synthetic MSRs. This ensures the hypercall page and SynIC facilities are set up before vCPUs attempt to use it.
--- accel/mshv/mshv-all.c | 7 +++++++ include/hw/hyperv/hvgdk_mini.h | 3 +++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 15 +++++++++++++++ 4 files changed, 26 insertions(+)
accel/mshv/mshv-all.c +7 -0
@@ -137,6 +137,13 @@ static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)static int mshv_load_cleanup (void *opaque) {MshvState *s = opaque; +int ret; + + ret =mshv_arch_set_partition_msrs (first_cpu); +if (ret <0 ) { +error_report ("Failed to set partition MSRs: %s" ,strerror (-ret)); +return -1 ; + }resume_vm (s->vm);
include/hw/hyperv/hvgdk_mini.h +3 -0
@@ -23,6 +23,9 @@#define HV_X64_MSR_APIC_FREQUENCY 0x40000023 typedef enum hv_register_name { + /* VP Management Registers */ + HV_REGISTER_INTERNAL_ACTIVITY_STATE = 0x00000004, + /* Pending Interruption Register */ HV_REGISTER_PENDING_INTERRUPTION = 0x00010002,
include/system/mshv_int.h +1 -0
@@ -84,6 +84,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs, size_t n_regs);int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu); +int mshv_arch_set_partition_msrs (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +15 -0
@@ -1115,6 +1115,21 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return 0 ; } +int mshv_arch_set_partition_msrs (const CPUState *cpu) +{ +CPUX86State *env = &X86_CPU (cpu)->env; +struct hv_register_assoc assocs[] = { + { .name =HV_REGISTER_GUEST_OS_ID , + .value.reg64 = env->msr_hv_guest_os_id }, + { .name =HV_REGISTER_REFERENCE_TSC , + .value.reg64 = env->msr_hv_tsc }, + { .name =HV_X64_REGISTER_HYPERCALL , + .value.reg64 = env->msr_hv_hypercall }, + }; + +return mshv_set_generic_regs (cpu, assocs,ARRAY_SIZE (assocs)); +} +void mshv_arch_amend_proc_features (union hv_partition_synthetic_processor_features *features) {
Signature · 1 line
2.34.1
PATCH 28/34 accel/mshv: migrate REFERENCE_TIME
In reply to Magnus KulkeThis is a partition-wide state that we use a dedicated handler for. It is stored as-is in the "mshv" savevm_live field. We might have to extend this later if we have more partition-wide state to roundtrip.
--- accel/mshv/mshv-all.c | 70 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-)
accel/mshv/mshv-all.c +69 -1
@@ -85,13 +85,81 @@ static int set_time_freeze(int vm_fd, int freeze)return 0 ; } +static int get_reference_time (int vm_fd,uint64_t *ref_time) +{ +int ret; +struct hv_input_get_partition_property in = {0 }; +struct hv_output_get_partition_property out = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code =HV_PARTITION_PROPERTY_REFERENCE_TIME ; + + args.code =HVCALL_GET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + args.out_sz =sizeof (out); + args.out_ptr = (uint64_t )&out; + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to get reference time" ); +return -1 ; + } + + *ref_time = out.property_value; +return 0 ; +} +static void mshv_save_state (QEMUFile *f,void *opaque) { -return ; +MshvState *s = opaque; +uint64_t ref_time; +int ret; + + ret =get_reference_time (s->vm, &ref_time); +if (ret <0 ) { +error_report ("Failed to get reference time for migration" ); +abort (); + } + +qemu_put_be64 (f, ref_time); +} + +static int set_reference_time (int vm_fd,uint64_t ref_time) +{ +int ret; +struct hv_input_set_partition_property in = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code =HV_PARTITION_PROPERTY_REFERENCE_TIME ; + in.property_value = ref_time; + + args.code =HVCALL_SET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to set reference time" ); +return -1 ; + } + +return 0 ; }static int mshv_load_state (QEMUFile *f,void *opaque,int version_id) { +MshvState *s = opaque; +uint64_t ref_time; +int ret; + + ref_time =qemu_get_be64 (f); + + ret =set_reference_time (s->vm, ref_time); +if (ret <0 ) { +error_report ("Failed to set reference time after migration" ); +return -1 ; + }return 0 ; }
Signature · 1 line
2.34.1
PATCH 29/34 target/i386/mshv: migrate pending ints/excs
In reply to Magnus KulkeWe use PENDING_INTERRUPTION, INTERRUPT_STATE, PENDING_EVENT hv registers to map and roundtrip from/to CPUX86State.
We ignore HV_REGISTER_PENDING_EVENT1 which represent events for nested virt contexts, as we don't support nested virt with MSHV currently.
--- include/hw/hyperv/hvgdk_mini.h | 3 + include/system/mshv_int.h | 13 +++ target/i386/mshv/mshv-cpu.c | 168 +++++++++++++++++++++++++++++++++ 3 files changed, 184 insertions(+)
include/hw/hyperv/hvgdk_mini.h +3 -0
@@ -28,6 +28,9 @@ typedef enum hv_register_name {/* Pending Interruption Register */ HV_REGISTER_PENDING_INTERRUPTION =0x00010002 , +HV_REGISTER_INTERRUPT_STATE =0x00010003 , +HV_REGISTER_PENDING_EVENT0 =0x00010004 , +HV_REGISTER_PENDING_EVENT1 =0x00010005 ,/* X64 User-Mode Registers */ HV_X64_REGISTER_RAX = 0x00020000,
include/system/mshv_int.h +13 -0
@@ -18,6 +18,19 @@struct mshv_get_set_vp_state ; +/* +* Interruption-type encoding, used by the hypervisor in +* hv_x64_pending_interruption_register.interruption_type +* See TLFS 6.0 section 7.9.2, p55 +* https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs +*/ +#define MSHV_HV_INTERRUPTION_TYPE_EXT_INT 0 +#define MSHV_HV_INTERRUPTION_TYPE_NMI 2 +#define MSHV_HV_INTERRUPTION_TYPE_HW_EXC 3 +#define MSHV_HV_INTERRUPTION_TYPE_SW_INT 4 +#define MSHV_HV_INTERRUPTION_TYPE_PRIV_SW_EXC 5 +#define MSHV_HV_INTERRUPTION_TYPE_SW_EXC 6 +typedef struct hyperv_message hv_message ; typedef struct MshvHvCallArgs {
target/i386/mshv/mshv-cpu.c +168 -0
186 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 30/34 target/i386: add de/compaction to xsave_helper
In reply to Magnus KulkeHyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.
- decompact_xsave_area(): converts compacted format to standard. XSTATE_BV is masked to host XCR0 since IA32_XSS is managed by the hypervisor.
- compact_xsave_area(): converts standard format back to compacted format. XCOMP_BV is set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.
Both functions use the host's CPUID leaf 0xD subleaves to determine component sizes, offsets, and alignment requirements.
There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (x86_cpu_expand_features).
--- target/i386/cpu.h | 2 + target/i386/xsave_helper.c | 255 +++++++++++++++++++++++++++++++++++++ 2 files changed, 257 insertions(+)
target/i386/cpu.h +2 -0
@@ -3033,6 +3033,8 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen);void x86_cpu_xsave_all_areas (X86CPU *cpu,void *buf,uint32_t buflen);uint32_t xsave_area_size (uint64_t mask,bool compacted);void x86_update_hflags (CPUX86State * env); +int decompact_xsave_area (const void *buf,size_t buflen,CPUX86State *env); +int compact_xsave_area (CPUX86State *env,void *buf,size_t buflen);static inline bool hyperv_feat_enabled (X86CPU *cpu,int feat) {
target/i386/xsave_helper.c +255 -0
264 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper
Quoted 26 lines · Magnus Kulke
HyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.
-
decompact_xsave_area(): converts compacted format to standard.XSTATE_BVis masked to host XCR0 sinceIA32_XSSis managed by the hypervisor.-
compact_xsave_area(): converts standard format back to compacted format.XCOMP_BVis set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.Both functions use the host's CPUID leaf
0xDsubleaves to determine component sizes, offsets, and alignment requirements.There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (
x86_cpu_expand_features).
Quoted 5 lines
Signed-off-by Magnus Kulke <[email protected]>---
target/i386/cpu.h| 2 +target/i386/xsave_helper.c| 255 +++++++++++++++++++++++++++++++++++++2 files changed, 257 insertions(+)
Quoted 288 lines
target/i386/cpu.h +2
-0@@ -3033,6 +3033,8 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen);void x86_cpu_xsave_all_areas (X86CPU *cpu,void *buf,uint32_t buflen);uint32_t xsave_area_size (uint64_t mask,bool compacted);void x86_update_hflags (CPUX86State * env); +int decompact_xsave_area (const void *buf,size_t buflen,CPUX86State *env); +int compact_xsave_area (CPUX86State *env,void *buf,size_t buflen);static inline bool hyperv_feat_enabled (X86CPU *cpu,int feat) {target/i386/xsave_helper.c +255
-0@@ -3,6 +3,7 @@* See the COPYING file in the top-level directory. */ #include "qemu/osdep.h" +#include "qemu/error-report.h" #include "cpu.h" @@ -293,3 +294,257 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen) }#endif } + +#define XSTATE_BV_IN_HDR offsetof(X86XSaveHeader, xstate_bv) +#define XCOMP_BV_IN_HDR offsetof(X86XSaveHeader, xcomp_bvo) + +typedef struct X86XSaveAreaView { +/* 512 bytes */ +X86LegacyXSaveArea legacy; +/* 64 bytes */ +X86XSaveHeader header; +/* ...followed by individual xsave areas */ +}X86XSaveAreaView ; + +#define XSAVE_XSTATE_BV_OFFSET offsetof(X86XSaveAreaView, header.xstate_bv) +#define XSAVE_XCOMP_BV_OFFSET offsetof(X86XSaveAreaView, header.xcomp_bv) +#define XSAVE_EXT_OFFSET (sizeof(X86LegacyXSaveArea) + \ + sizeof(X86XSaveHeader)) + +/** +* decompact_xsave_area - Convert compacted XSAVE format to standard format +* @buf: Source buffer containing compacted XSAVE data +* @buflen: Size of source buffer +* @env: CPU state where the standard format buffer will be written to +* +* Accelerator backends like MSHV might return XSAVE state in compacted format +* (XSAVEC). The state components have to be packed contiguously without gaps. +* The XSAVE qemu buffers are in standard format where each component has a +* fixed offset. +* +* Returns: 0 on success, negative errno on failure +*/ +int decompact_xsave_area (const void *buf,size_t buflen,CPUX86State *env) +{ +uint64_t compacted_xstate_bv, compacted_xcomp_bv, compacted_layout_bv; +uint64_t xsave_offset, *xcomp_bv; +size_t i; +uint32_t eax, ebx, ecx, edx; +uint32_t size, dst_off; +bool align64; +uint64_t guest_xcr0, *xstate_bv; + + compacted_xstate_bv = *(uint64_t *)(buf +XSAVE_XSTATE_BV_OFFSET ); + compacted_xcomp_bv = *(uint64_t *)(buf +XSAVE_XCOMP_BV_OFFSET ); + +/* This function only handles compacted format (bit 63 set) */ +assert ((compacted_xcomp_bv >>63 ) &1 ); + +/* Low bits of XCOMP_BV describe which components are in the layout */ + compacted_layout_bv = compacted_xcomp_bv & ~(1ULL <<63 ); + +/* Zero out buffer, then copy legacy region (FP + SSE) and header as-is */ +memset (env->xsave_buf,0 , env->xsave_buf_len); +memcpy (env->xsave_buf, buf,XSAVE_EXT_OFFSET ); + +/* +* We mask XSTATE_BV with the guest's supported XCR0 because: +* 1. Supervisor state (IA32_XSS) is hypervisor-managed, we don't use +* this state for migration. +* 2. Features disabled at partition creation (e.g. AMX) must be excluded +*/ + guest_xcr0 = ((uint64_t )env->features[FEAT_XSAVE_XCR0_HI ] <<32 ) | + env->features[FEAT_XSAVE_XCR0_LO ]; + xstate_bv = (uint64_t *)(env->xsave_buf +XSAVE_XSTATE_BV_OFFSET ); + *xstate_bv &= guest_xcr0; + +/* Clear bit 63 - output is standard format, not compacted */ + xcomp_bv = (uint64_t *)(env->xsave_buf +XSAVE_XCOMP_BV_OFFSET ); + *xcomp_bv = *xcomp_bv & ~(1ULL <<63 ); + +/* +* Process each extended state component in the compacted layout. +* Components 0 and 1 (FP and SSE) are in the legacy region, so we +* start at component 2. For each component: +* - Calculate its offset in the compacted source (contiguous layout) +* - Get its fixed offset in the standard destination from CPUID +* - Copy if the component has non-init state (bit set in XSTATE_BV) +*/ + xsave_offset =XSAVE_EXT_OFFSET ; +for (i =2 ; i <63 ; i++) { +if (((compacted_layout_bv >> i) &1 ) ==0 ) { +continue ; + } + +/* Query guest CPUID for this component's size and standard offset */ +cpu_x86_cpuid (env,0xD , i, &eax, &ebx, &ecx, &edx); + + size = eax; + dst_off = ebx; + align64 = (ecx & (1u <<1 )) !=0 ; + +/* Component is in the layout but unknown to the guest CPUID model */ +if (size ==0 ) { +/* +* The hypervisor might expose a component that has no +* representation in the guest CPUID model. We query the host to +* retrieve the size of the component, so we can skip over it. +*/ +host_cpuid (0xD , i, &eax, &ebx, &ecx, &edx); + size = eax; + align64 = (ecx & (1u <<1 )) !=0 ; +if (size ==0 ) { +error_report ("xsave component %zu: size unknown to both " +"guest and host CPUID" , i); +return -EINVAL ; + } + +if (align64) { + xsave_offset =QEMU_ALIGN_UP (xsave_offset,64 ); + } + +if (xsave_offset + size > buflen) { +error_report ("xsave component %zu overruns source buffer: " +"offset=%zu size=%u buflen=%zu" , + i, xsave_offset, size, buflen); +return -E2BIG ; + } + + xsave_offset += size; +continue ; + } + +if (align64) { + xsave_offset =QEMU_ALIGN_UP (xsave_offset,64 ); + } + +if ((xsave_offset + size) > buflen) { +error_report ("xsave component %zu overruns source buffer: " +"offset=%zu size=%u buflen=%zu" , + i, xsave_offset, size, buflen); +return -E2BIG ; + } + +if ((dst_off + size) > env->xsave_buf_len) { +error_report ("xsave component %zu overruns destination buffer: " +"offset=%u size=%u buflen=%zu" , + i, dst_off, size, (size_t )env->xsave_buf_len); +return -E2BIG ; + } + +/* Copy components marked present in XSTATE_BV to guest model */ +if (((compacted_xstate_bv >> i) &1 ) !=0 ) { +memcpy (env->xsave_buf + dst_off, buf + xsave_offset, size); + } + + xsave_offset += size; + } + +return 0 ; +} + +/** +* compact_xsave_area - Convert standard XSAVE format to compacted format +* @env: CPU state containing the standard format XSAVE buffer +* @buf: Destination buffer for compacted XSAVE data (to send to hypervisor) +* @buflen: Size of destination buffer +* +* Accelerator backends like MSHV might expect XSAVE state in compacted format +* (XSAVEC). The state components are packed contiguously without gaps. +* The XSAVE qemu buffers are in standard format where each component has a +* fixed offset. +* +* This function converts from standard to compacted format, it accepts a +* pre-allocated destination buffer of sufficient size, it is the +* responsibility of the caller to ensure the buffer is big enough. +* +* Returns: total size of compacted XSAVE data written to @buf +*/ +int compact_xsave_area (CPUX86State *env,void *buf,size_t buflen) +{ +uint64_t *xcomp_bv; +size_t i; +uint32_t eax, ebx, ecx, edx; +uint32_t size, src_off; +bool align64; +size_t compact_offset; +uint64_t host_xcr0_mask, guest_xcr0; + +/* Zero out buffer, then copy legacy region (FP + SSE) and header as-is */ +memset (buf,0 , buflen); +memcpy (buf, env->xsave_buf,XSAVE_EXT_OFFSET ); + +/* +* Set XCOMP_BV to indicate compacted format (bit 63) and which +* components are in the layout. +* +* We must explicitly set XCOMP_BV because x86_cpu_xsave_all_areas() +* produces standard format with XCOMP_BV=0 (buffer is zeroed and only +* XSTATE_BV is set in the header). +* +* XCOMP_BV must reflect the partition's XSAVE capability, not the +* guest's current XCR0 (env->xcr0). These differ b/c: +* - A guest's XCR0 is what the guest OS has enabled via XSETBV +* - The partition's XCR0 mask is the hypervisor's save/restore capability +* +* The hypervisor uses XSAVES which saves based on its capability, so the +* XCOMP_BV value in the buffer we send back must match that capability. +* +* We intersect the host XCR0 with the guest's supported XCR0 features +* (FEAT_XSAVE_XCR0_*) so that features disabled at partition creation +* (e.g. AMX) are excluded from the compacted layout. +*/ +host_cpuid (0xD ,0 , &eax, &ebx, &ecx, &edx); + host_xcr0_mask = ((uint64_t )edx <<32 ) | eax; + guest_xcr0 = ((uint64_t )env->features[FEAT_XSAVE_XCR0_HI ] <<32 ) | + env->features[FEAT_XSAVE_XCR0_LO ]; + host_xcr0_mask &= guest_xcr0; + xcomp_bv = buf +XSAVE_XCOMP_BV_OFFSET ; + *xcomp_bv = host_xcr0_mask | (1ULL <<63 ); + +/* +* Process each extended state component in the host's XCR0. +* The compacted layout must match XCOMP_BV (host capability). +* +* For each component: +* - Get its size and standard offset from host CPUID +* - Apply 64-byte alignment if required +* - Copy data only if guest has this component (bit set in env->xcr0) +* - Always advance offset to maintain correct layout +*/ + compact_offset =XSAVE_EXT_OFFSET ; +for (i =2 ; i <63 ; i++) { +if (!((host_xcr0_mask >> i) &1 )) { +continue ; + } + +/* Query host CPUID for this component's size and standard offset */ +host_cpuid (0xD , i, &eax, &ebx, &ecx, &edx); + size = eax; + src_off = ebx; + align64 = (ecx >>1 ) &1 ; + +if (size ==0 ) { +/* Component in host xcr0 but unknown - shouldn't happen */ +continue ; + } + +/* Apply 64-byte alignment if required by this component */ +if (align64) { + compact_offset =QEMU_ALIGN_UP (compact_offset,64 ); + } + +/* +* Only copy data if guest has this component enabled in XCR0. +* Otherwise the component remains zeroed (init state), but we +* still advance the offset to maintain the correct layout. +*/ +if ((env->xcr0 >> i) &1 ) { +memcpy (buf + compact_offset, env->xsave_buf + src_off, size); + } + + compact_offset += size; + } + +return compact_offset; +}Signature · 1 line
2.34.1
Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper
In reply to Magnus KulkeQuoted 21 lines · Magnus Kulke
HyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.
-
decompact_xsave_area(): converts compacted format to standard.XSTATE_BVis masked to host XCR0 sinceIA32_XSSis managed by the hypervisor.-
compact_xsave_area(): converts standard format back to compacted format.XCOMP_BVis set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.Both functions use the host's CPUID leaf
0xDsubleaves to determine component sizes, offsets, and alignment requirements.
Quoted 6 lines
There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (
x86_cpu_expand_features).Signed-off-by Magnus Kulke <[email protected]>
Hi,
This patch has a tiny problem.
#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
void *xsave_buf;uint32_t xsave_buf_len;
#endif
With the "#if defined(CONFIG_KVM) || defined(CONFIG_HVF)” removed, that fixes a TCG only QEMU build.
Was also able to test that this works as expected on WHPX.
Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper
Quoted 10 lines · Mohamed Mediouni
Quoted 3 lines · Magnus Kulke
This patch has a tiny problem.
#if
defined(CONFIG_KVM)||defined(CONFIG_HVF)
void*xsave_buf;uint32_txsave_buf_len;
#endif
Quoted 4 lines
With the "#if
defined(CONFIG_KVM)|| defined(CONFIG_HVF)” removed, that fixes a TCG only QEMU build.Was also able to test that this works as expected on WHPX.
thx, that's right. we can also change the signature to pass xsave_buf and xsave_buf_len as params, so accelerators which do not have xsave buffers do not choke on this, it's probably less invasive rather than including xsave_buf fields for all accelerators.
PATCH 31/34 target/i386/mshv: migrate XSAVE state
In reply to Magnus KulkeWe implement fn's that roundtrip XSAVE state in migration. We are using the xsave_helper routines to move individual components from CPUX86State to an xsave_buf and then we have to compact the buffer to XSAVEC format, which is what the hypervisor expects.
And the same applies in the other direction for restoring state from the hypervisor.
--- target/i386/cpu.h | 2 +- target/i386/mshv/mshv-cpu.c | 100 +++++++++++++++++++++++++++++++++++- 2 files changed, 100 insertions(+), 2 deletions(-)
target/i386/cpu.h +1 -1
@@ -2272,7 +2272,7 @@ typedef struct CPUArchState {int64_t user_tsc_khz;/* for sanity check only */ uint64_t apic_bus_freq;uint64_t tsc; -#if defined(CONFIG_KVM ) || defined(CONFIG_HVF ) +#if defined(CONFIG_KVM ) || defined(CONFIG_HVF ) || defined(CONFIG_MSHV )void *xsave_buf;uint32_t xsave_buf_len;#endif
target/i386/mshv/mshv-cpu.c +99 -1
138 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 32/34 target/i386/mshv: reconstruct hflags after load
In reply to Magnus Kulkehflags is a cached bitmap derived from standard and special regs. We want to reconstruct this state after regs and sregs have been read from the hypervisor, similar to how it's one in other accelerators.
--- target/i386/mshv/mshv-cpu.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
target/i386/mshv/mshv-cpu.c +13 -0
@@ -814,6 +814,16 @@ static int set_vcpu_events(const CPUState *cpu)return 0 ; } +static int update_hflags (CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); +CPUX86State *env = &x86cpu->env; + +x86_update_hflags (env); + +return 0 ; +} +int mshv_arch_load_vcpu_state (CPUState *cpu) {int ret;
@@ -828,6 +838,9 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } +/* INVARIANT: hflags are derived from regs+sregs, need to get both first */ +update_hflags (cpu); + ret =get_xc_reg (cpu);if (ret <0 ) {return ret;
Signature · 1 line
2.34.1
PATCH 33/34 target/i386/mshv: migrate MP_STATE
In reply to Magnus KulkeMSHV's "internal activity state" roughly maps to QEMU's env->mp_state and cpu->halted states that describe state of APs in a guest.
We don't invoke set_mp_state as part of store_vcpu_state() b/c we would put all BSP + APs in a RUNNABLE (0) state immediately, breaking SMP boot
Instead we store the mp state as part of the load_cleanup() routine after a migration.
--- accel/mshv/mshv-all.c | 10 +++++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 80 +++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+)
accel/mshv/mshv-all.c +10 -0
@@ -205,6 +205,7 @@ static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)static int mshv_load_cleanup (void *opaque) {MshvState *s = opaque; +CPUState *cpu;int ret; ret =mshv_arch_set_partition_msrs (first_cpu);
@@ -213,6 +214,15 @@ static int mshv_load_cleanup(void *opaque)return -1 ; } +CPU_FOREACH (cpu) { + ret =mshv_arch_set_mp_state (cpu); +if (ret <0 ) { +error_report ("Failed to set mp state for vCPU %d: %s" , + cpu->cpu_index,strerror (-ret)); +return -1 ; + } + } +resume_vm (s->vm);return 0 ;
include/system/mshv_int.h +1 -0
@@ -98,6 +98,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu);int mshv_arch_set_partition_msrs (const CPUState *cpu); +int mshv_arch_set_mp_state (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +80 -0
98 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH 34/34 accel/mshv: enable dirty page tracking
In reply to Magnus KulkeThis change introduces the functions required to perform dirty page tracking to speed up migrations. We are using the sync, global_start, and global_stop hooks.
The sync is implemented in batches.
Before we can disable the dirty page tracking we have to set all dirty bits.
--- accel/mshv/mem.c | 211 ++++++++++++++++++++++++++++++++++++++ accel/mshv/mshv-all.c | 3 + include/system/mshv_int.h | 5 + 3 files changed, 219 insertions(+)
accel/mshv/mem.c +211 -0
@@ -12,10 +12,13 @@#include "qemu/osdep.h" #include "qemu/error-report.h" +#include "qapi/error.h" #include "linux/mshv.h" #include "system/address-spaces.h" #include "system/mshv.h" #include "system/mshv_int.h" +#include "hw/hyperv/hvhdk_mini.h" +#include "system/physmem.h" #include "exec/memattrs.h" #include <sys/ioctl.h> #include "trace.h"
@@ -211,3 +214,211 @@ void mshv_set_phys_mem(MshvMemoryListener *mml, MemoryRegionSection *section,abort (); } } + +static int enable_dirty_page_tracking (int vm_fd) +{ +int ret; +struct hv_input_set_partition_property in = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code =HV_PARTITION_PROPERTY_GPA_PAGE_ACCESS_TRACKING ; + in.property_value =1 ; + + args.code =HVCALL_SET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to enable dirty page tracking: %s" , +strerror (errno)); +return -1 ; + } + +return 0 ; +} + +/* +* Retrieve dirty page bitmap for a GPA range, clearing the dirty bits +* atomically. Large ranges are handled in batches. +*/ +static int get_dirty_log (int vm_fd,uint64_t base_pfn,uint64_t page_count, +unsigned long *bitmap,size_t bitmap_size) +{ +uint64_t batch, bitmap_offset, completed =0 ; +struct mshv_gpap_access_bitmap args = {0 }; +int ret; + +QEMU_BUILD_BUG_ON (MSHV_DIRTY_PAGES_BATCH_SIZE %BITS_PER_LONG !=0 ); +assert (bitmap_size >=ROUND_UP (page_count,BITS_PER_LONG ) /8 ); + +while (completed < page_count) { + batch =MIN (MSHV_DIRTY_PAGES_BATCH_SIZE , page_count - completed); + bitmap_offset = completed /BITS_PER_LONG ; + + args.access_type =MSHV_GPAP_ACCESS_TYPE_DIRTY ; + args.access_op =MSHV_GPAP_ACCESS_OP_CLEAR ; + args.page_count = batch; + args.gpap_base = base_pfn + completed; + args.bitmap_ptr = (uint64_t )(bitmap + bitmap_offset); + + ret =ioctl (vm_fd,MSHV_GET_GPAP_ACCESS_BITMAP , &args); +if (ret <0 ) { +error_report ("Failed to get dirty log (base_pfn=0x%" PRIx64 +" batch=%" PRIu64"): %s" , + base_pfn + completed, batch,strerror (errno)); +return -1 ; + } + completed += batch; + } + +return 0 ; +} + +bool mshv_log_global_start (MemoryListener *listener,Error **errp) +{ +int ret; + + ret =enable_dirty_page_tracking (mshv_state->vm); +if (ret <0 ) { +error_setg_errno (errp, -ret,"Failed to enable dirty page tracking" ); +return false; + } +return true; +} + +static int disable_dirty_page_tracking (int vm_fd) +{ +int ret; +struct hv_input_set_partition_property in = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code =HV_PARTITION_PROPERTY_GPA_PAGE_ACCESS_TRACKING ; + in.property_value =0 ; + + args.code =HVCALL_SET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to disable dirty page tracking: %s" , +strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static int set_dirty_pages (int vm_fd,uint64_t base_pfn,uint64_t page_count) +{ +uint64_t batch, completed =0 ; +unsigned long bitmap[MSHV_DIRTY_PAGES_BATCH_SIZE /BITS_PER_LONG ]; +struct mshv_gpap_access_bitmap args = {0 }; +int ret; + +while (completed < page_count) { + batch =MIN (MSHV_DIRTY_PAGES_BATCH_SIZE , page_count - completed); + + args.access_type =MSHV_GPAP_ACCESS_TYPE_DIRTY ; + args.access_op =MSHV_GPAP_ACCESS_OP_SET ; + args.page_count = batch; + args.gpap_base = base_pfn + completed; + args.bitmap_ptr = (uint64_t )bitmap; + + ret =ioctl (vm_fd,MSHV_GET_GPAP_ACCESS_BITMAP , &args); +if (ret <0 ) { +error_report ("Failed to set dirty pages (base_pfn=0x%" PRIx64 +" batch=%" PRIu64"): %s" , + base_pfn + completed, batch,strerror (errno)); +return -1 ; + } + completed += batch; + } + +return 0 ; +} + +static bool set_dirty_bits_cb (Int128 start,Int128 len,const MemoryRegion *mr, +hwaddr offset_in_region,void *opaque) +{ +int ret, *errp = opaque; +hwaddr gpa, size; +uint64_t page_count, base_pfn; + + gpa =int128_get64 (start); + size =int128_get64 (len); + page_count = size >>MSHV_PAGE_SHIFT ; + base_pfn = gpa >>MSHV_PAGE_SHIFT ; + +if (!mr->ram || mr->readonly) { +return false; + } + +if (page_count ==0 ) { +return false; + } + + ret =set_dirty_pages (mshv_state->vm, base_pfn, page_count); + +/* true aborts the iteration, which is what we want if there's an error */ +if (ret <0 ) { + *errp = ret; +return true; + } + +return false; +} + +void mshv_log_global_stop (MemoryListener *listener) +{ +int err =0 ; +/* MSHV requires all dirty bits to be set before disabling tracking. */ +FlatView *fv =address_space_to_flatview (&address_space_memory); +flatview_for_each_range (fv, set_dirty_bits_cb, &err); + +if (err <0 ) { +error_report ("Failed to set dirty bits before disabling tracking" ); + } + +disable_dirty_page_tracking (mshv_state->vm); +} + +void mshv_log_sync (MemoryListener *listener,MemoryRegionSection *section) +{ +hwaddr size, start_addr, mr_offset; +uint64_t page_count, base_pfn; +size_t bitmap_size; +unsigned long *bitmap; +ram_addr_t ram_addr; +int ret; +MemoryRegion *mr = section->mr; + +if (!memory_region_is_ram (mr) ||memory_region_is_rom (mr)) { +return ; + } + + size =align_section (section, &start_addr); +if (!size) { +return ; + } + + page_count = size >>MSHV_PAGE_SHIFT ; + base_pfn = start_addr >>MSHV_PAGE_SHIFT ; + bitmap_size =ROUND_UP (page_count,BITS_PER_LONG ) /8 ; + bitmap =g_malloc0 (bitmap_size); + + ret =get_dirty_log (mshv_state->vm, base_pfn, page_count, bitmap, + bitmap_size); +if (ret <0 ) { +g_free (bitmap); +return ; + } + + mr_offset = section->offset_within_region + start_addr - + section->offset_within_address_space; + ram_addr =memory_region_get_ram_addr (mr) + mr_offset; + +physical_memory_set_dirty_lebitmap (bitmap, ram_addr, page_count); +g_free (bitmap); +}
accel/mshv/mshv-all.c +3 -0
@@ -546,6 +546,9 @@ static MemoryListener mshv_memory_listener = { .region_del = mem_region_del, .eventfd_add = mem_ioeventfd_add, .eventfd_del = mem_ioeventfd_del, + .log_sync = mshv_log_sync, + .log_global_start = mshv_log_global_start, + .log_global_stop = mshv_log_global_stop, };static MemoryListener mshv_io_listener = {
include/system/mshv_int.h +5 -0
@@ -31,6 +31,8 @@ struct mshv_get_set_vp_state;#define MSHV_HV_INTERRUPTION_TYPE_PRIV_SW_EXC 5#define MSHV_HV_INTERRUPTION_TYPE_SW_EXC 6 +#define MSHV_DIRTY_PAGES_BATCH_SIZE 0x10000 +typedef struct hyperv_message hv_message ; typedef struct MshvHvCallArgs {
@@ -128,6 +130,9 @@ int mshv_guest_mem_write(uint64_t gpa, const uint8_t *data, uintptr_t size, bool is_secure_mode);void mshv_set_phys_mem (MshvMemoryListener *mml,MemoryRegionSection *section,bool add); +void mshv_log_sync (MemoryListener *listener,MemoryRegionSection *section); +bool mshv_log_global_start (MemoryListener *listener,Error **errp); +void mshv_log_global_stop (MemoryListener *listener);/* msr */ int mshv_init_msrs (const CPUState *cpu);
Signature · 1 line
2.34.1
Paolo Bonzini
maintainerI merged everything except:
- target/i386/mshv: migrate LAPIC state (*)
- target/i386/mshv: migrate Synic SINT MSRs
- target/i386/mshv: migrate SIMP and SIEFP state
- target/i386/mshv: migrate STIMER state
- accel/mshv: introduce SaveVMHandler (*)
- accel/mshv: write synthetic MSRs after migration
- accel/mshv: migrate REFERENCE_TIME
- target/i386/mshv: migrate MP_STATE
The ones marked (*) were the important ones that I'll reply to, the rest are collateral damage.
I also removed TSC_DEADLINE from the MSRs patch, since it's more of a local APIC register and it has a dependency on "migrate LAPIC state".
Paolo