Skip to contentSkip to conversation

Add migration support to the MSHV accelerator

PATCH In review Magnus Kulke 3 participants 54 messages Apr 17 – Jul 6

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add migration support to the MSHV accelerator

Hey all, This is a patch series for live migration support in the MSHV accelerator. Since this is somewhat invasive and touches various parts outside of the accel's folder hierarchies I'm sending an RFC series to collect early feedback.

This patches are sent in sent in isolation, but for LM to be fully functional we will require require the recently submitted patch series "Support QEMU cpu models in MSHV accelerator" (at v6 currently) to be merged, since on some hosts we will need the CPUID infra to disable features that we currently are not able to migrate yet (e.g. AMX tiles).

In this series we perform some preperatory refactorings and introduce new abstractions where required, particulary for irqchips and MSR logic. We also have to introduce some generic logic for XSAVE de/compaction to allow migration of XSAVE state.

Note: there is some pending optimization to avoid a buffer copy and rework existing xsave helper code to be generic over a compacted or standard layout. We'll address this in a later revision or in a discrete patch.

The guest state components that are covered by migration are:

- standard regs
- special regs
- xcr0
- (legacy) FPU regs
- XSAVE
- LAPIC
- MSRs
- SynIC state (SIMP, SIEFP, STIMER)
- pending interrupts/exceptions
- MP state (AP cpu modes)

Finally, routines for dirty-page tracking to reduce migration downtime have beend added and integrated in the respective hooks.

best,

magnus

Changes since RFC:

- Added CET SS/IBT MSR migration
- Assert 64bit padding on CPUX86State->sysenter_cs statically

Magnus Kulke (34):
target/i386/mshv: use arch_load/store_reg fns target/i386/mshv: use generic FPU/xcr0 state target/i386/mshv: impl init/load/store_vcpu_state accel/accel-irq: add AccelRouteChange abstraction accel/accel-irq: add generic begin_route_changes accel/accel-irq: add generic commit_route_changes accel/mshv: add irq_routes to state accel/mshv: update s->irq_routes in add_msi_route accel/mshv: update s->irq_routes in update_msi_route accel/mshv: update s->irq_routes in release_virq accel/mshv: use s->irq_routes in commit_routes accel/mshv: reserve ioapic routes on s->irq_routes accel/mshv: remove redundant msi controller target/i386/mshv: move apic logic into own file target/i386/mshv: remove redundant apic helpers target/i386/mshv: migrate LAPIC state target/i386/mshv: move msr code to arch accel/mshv: store partition proc features target/i386/mshv: expose msvh_get_generic_regs target/i386/mshv: migrate MSRs target/i386/mshv: migrate MTRR MSRs target/i386/mshv: migrate Synic SINT MSRs target/i386/mshv: migrate CET/SS MSRs target/i386/mshv: migrate SIMP and SIEFP state target/i386/mshv: migrate STIMER state accel/mshv: introduce SaveVMHandler accel/mshv: write synthetic MSRs after migration accel/mshv: migrate REFERENCE_TIME target/i386/mshv: migrate pending ints/excs target/i386: add de/compaction to xsave_helper target/i386/mshv: migrate XSAVE state target/i386/mshv: reconstruct hflags after load target/i386/mshv: migrate MP_STATE accel/mshv: enable dirty page tracking

 accel/accel-irq.c               |  41 +-
 accel/kvm/kvm-all.c             |   6 +-
 accel/mshv/irq.c                | 360 ++++++------
 accel/mshv/mem.c                | 211 +++++++
 accel/mshv/meson.build          |   1 -
 accel/mshv/mshv-all.c           | 243 +++++++-
 accel/mshv/msr.c                | 375 -------------
 accel/stubs/kvm-stub.c          |   2 +-
 accel/stubs/mshv-stub.c         |   6 +-
 hw/intc/apic_common.c           |   3 +
 hw/misc/ivshmem-pci.c           |   8 +-
 hw/vfio/pci.c                   |  11 +-
 hw/virtio/virtio-pci.c          |   3 +-
 include/accel/accel-route.h     |  17 +
 include/hw/hyperv/hvgdk_mini.h  |  33 ++
 include/hw/hyperv/hvhdk.h       | 150 +++++
 include/hw/i386/apic_internal.h |   5 +
 include/system/accel-irq.h      |   6 +-
 include/system/kvm.h            |  23 +-
 include/system/mshv.h           |  15 +-
 include/system/mshv_int.h       |  89 +--
 target/i386/cpu.h               |  14 +-
 target/i386/kvm/kvm.c           |   5 +-
 target/i386/machine.c           |  46 ++
 target/i386/mshv/meson.build    |   3 +
 target/i386/mshv/mshv-apic.c    |  78 +++
 target/i386/mshv/mshv-cpu.c     | 958 +++++++++++++++++++++++---------
 target/i386/mshv/msr.c          | 467 ++++++++++++++++
 target/i386/mshv/synic.c        | 206 +++++++
 target/i386/xsave_helper.c      | 255 +++++++++
 30 files changed, 2684 insertions(+), 956 deletions(-)
 delete mode 100644 accel/mshv/msr.c
 create mode 100644 include/accel/accel-route.h
 create mode 100644 target/i386/mshv/mshv-apic.c
 create mode 100644 target/i386/mshv/msr.c
 create mode 100644 target/i386/mshv/synic.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/34 target/i386/mshv: use arch_load/store_reg fns

Improved consistency around the naming of load/store register fn's. this is required since we want to roundtrip more registers in a migration than what's currently required for MMIO emulation.

---
 accel/mshv/mshv-all.c       |  2 +-
 include/system/mshv_int.h   |  6 ++---
 target/i386/mshv/mshv-cpu.c | 52 ++++++++++++++-----------------------
 3 files changed, 23 insertions(+), 37 deletions(-)
accel/mshv/mshv-all.c +1 -1
index d4cc7f5371..7c0eb68a5b 100644
@@ -650,7 +650,7 @@ static void mshv_cpu_synchronize_pre_loadvm(CPUState *cpu)
 static void do_mshv_cpu_synchronize(CPUState *cpu, run_on_cpu_data arg)
 {
     if (!cpu->accel->dirty) {
-        int ret = mshv_load_regs(cpu);
+        int ret = mshv_arch_load_regs(cpu);
         if (ret < 0) {
             error_report("Failed to load registers for vcpu %d",
                          cpu->cpu_index);
include/system/mshv_int.h +2 -4
index 35386c422f..a142dd241a 100644
@@ -82,11 +82,9 @@ void mshv_init_mmio_emu(void);
 int mshv_create_vcpu(int vm_fd, uint8_t vp_index, int *cpu_fd);
 void mshv_remove_vcpu(int vm_fd, int cpu_fd);
 int mshv_configure_vcpu(const CPUState *cpu, const MshvFPU *fpu, uint64_t xcr0);
-int mshv_get_standard_regs(CPUState *cpu);
-int mshv_get_special_regs(CPUState *cpu);
 int mshv_run_vcpu(int vm_fd, CPUState *cpu, hv_message *msg, MshvVmExit *exit);
-int mshv_load_regs(CPUState *cpu);
-int mshv_store_regs(CPUState *cpu);
+int mshv_arch_load_regs(CPUState *cpu);
+int mshv_arch_store_regs(CPUState *cpu);
 int mshv_set_generic_regs(const CPUState *cpu, const hv_register_assoc *assocs,
                           size_t n_regs);
 int mshv_arch_put_registers(const CPUState *cpu);
target/i386/mshv/mshv-cpu.c +20 -32

128 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/34 target/i386/mshv: use generic FPU/xcr0 state

In reply to Magnus Kulke

Instead of using an mshv-specific FPU state representation we switch to the generic i386 representation of the registers.

---
 include/system/mshv_int.h   | 15 +-------
 target/i386/mshv/mshv-cpu.c | 76 ++++++++++++++++++++++---------------
 2 files changed, 47 insertions(+), 44 deletions(-)
include/system/mshv_int.h +1 -14
index a142dd241a..e3d1867a77 100644
@@ -58,19 +58,6 @@ typedef struct MshvMsiControl {
 #define mshv_vcpufd(cpu) (cpu->accel->cpufd)
 
 /* cpu */
-typedef struct MshvFPU {
-    uint8_t fpr[8][16];
-    uint16_t fcw;
-    uint16_t fsw;
-    uint8_t ftwx;
-    uint8_t pad1;
-    uint16_t last_opcode;
-    uint64_t last_ip;
-    uint64_t last_dp;
-    uint8_t xmm[16][16];
-    uint32_t mxcsr;
-    uint32_t pad2;
-} MshvFPU;
 
 typedef enum MshvVmExit {
     MshvVmExitIgnore   = 0,
@@ -81,7 +68,7 @@ typedef enum MshvVmExit {
 void mshv_init_mmio_emu(void);
 int mshv_create_vcpu(int vm_fd, uint8_t vp_index, int *cpu_fd);
 void mshv_remove_vcpu(int vm_fd, int cpu_fd);
-int mshv_configure_vcpu(const CPUState *cpu, const MshvFPU *fpu, uint64_t xcr0);
+int mshv_configure_vcpu(const CPUState *cpu);
 int mshv_run_vcpu(int vm_fd, CPUState *cpu, hv_message *msg, MshvVmExit *exit);
 int mshv_arch_load_regs(CPUState *cpu);
 int mshv_arch_store_regs(CPUState *cpu);
target/i386/mshv/mshv-cpu.c +46 -30

160 lines not on this page · raw message

Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/34 target/i386/mshv: impl init/load/store_vcpu_state

In reply to Magnus Kulke

In migration we will handle more than registers, so we rework the routines that were used to load & store CPU registers from/to the hypervisor into more explicit init/load/store_vcpu_state() functions that can be called from the appropriate hooks.

load/store_regs() still exists for the purpose of MMIO emulation, but it will only address standard and special x86 registers.

Functions to retrieve FPU and XCR0 state from the hypervsisor have been introduced.

MSR and APIC state covered are covered only as part of init_vcpu(). They are not yet part of the load/store routines.

---
 accel/mshv/mshv-all.c       |  10 +-
 include/system/mshv_int.h   |   5 +-
 target/i386/mshv/mshv-cpu.c | 354 ++++++++++++++++++------------------
 3 files changed, 185 insertions(+), 184 deletions(-)
accel/mshv/mshv-all.c +5 -5
index 7c0eb68a5b..04d248fe1d 100644
@@ -400,13 +400,13 @@ static int mshv_init_vcpu(CPUState *cpu)
     int ret;
 
     cpu->accel = g_new0(AccelCPUState, 1);
-    mshv_arch_init_vcpu(cpu);
 
     ret = mshv_create_vcpu(vm_fd, vp_index, &cpu->accel->cpufd);
     if (ret < 0) {
         return -1;
     }
 
+    mshv_arch_init_vcpu(cpu);
     cpu->accel->dirty = true;
 
     return 0;
@@ -488,7 +488,7 @@ static int mshv_cpu_exec(CPUState *cpu)
 
     do {
         if (cpu->accel->dirty) {
-            ret = mshv_arch_put_registers(cpu);
+            ret = mshv_arch_store_vcpu_state(cpu);
             if (ret) {
                 error_report("Failed to put registers after init: %s",
                               strerror(-ret));
@@ -610,7 +610,7 @@ static void mshv_start_vcpu_thread(CPUState *cpu)
 static void do_mshv_cpu_synchronize_post_init(CPUState *cpu,
                                               run_on_cpu_data arg)
 {
-    int ret = mshv_arch_put_registers(cpu);
+    int ret = mshv_arch_store_vcpu_state(cpu);
     if (ret < 0) {
         error_report("Failed to put registers after init: %s", strerror(-ret));
         abort();
@@ -626,7 +626,7 @@ static void mshv_cpu_synchronize_post_init(CPUState *cpu)
 
 static void mshv_cpu_synchronize_post_reset(CPUState *cpu)
 {
-    int ret = mshv_arch_put_registers(cpu);
+    int ret = mshv_arch_store_vcpu_state(cpu);
     if (ret) {
         error_report("Failed to put registers after reset: %s",
                      strerror(-ret));
@@ -650,7 +650,7 @@ static void mshv_cpu_synchronize_pre_loadvm(CPUState *cpu)
 static void do_mshv_cpu_synchronize(CPUState *cpu, run_on_cpu_data arg)
 {
     if (!cpu->accel->dirty) {
-        int ret = mshv_arch_load_regs(cpu);
+        int ret = mshv_arch_load_vcpu_state(cpu);
         if (ret < 0) {
             error_report("Failed to load registers for vcpu %d",
                          cpu->cpu_index);
include/system/mshv_int.h +2 -3
index e3d1867a77..70631ca6ba 100644
@@ -70,11 +70,10 @@ int mshv_create_vcpu(int vm_fd, uint8_t vp_index, int *cpu_fd);
 void mshv_remove_vcpu(int vm_fd, int cpu_fd);
 int mshv_configure_vcpu(const CPUState *cpu);
 int mshv_run_vcpu(int vm_fd, CPUState *cpu, hv_message *msg, MshvVmExit *exit);
-int mshv_arch_load_regs(CPUState *cpu);
-int mshv_arch_store_regs(CPUState *cpu);
 int mshv_set_generic_regs(const CPUState *cpu, const hv_register_assoc *assocs,
                           size_t n_regs);
-int mshv_arch_put_registers(const CPUState *cpu);
+int mshv_arch_store_vcpu_state(const CPUState *cpu);
+int mshv_arch_load_vcpu_state(CPUState *cpu);
 void mshv_arch_init_vcpu(CPUState *cpu);
 void mshv_arch_destroy_vcpu(CPUState *cpu);
 void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +178 -176

486 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw

PATCH 4/34 accel/accel-irq: add AccelRouteChange abstraction

In reply to Magnus Kulke

The accelerated irqchip routines use a record of changes to batch changes when programming routes.

Currently this mechanism is coupled to the KVM accelerator, this change introduces an abstraction that replaces KVMRouteChange and keeps a pointer to an abstract AccelState instead of the concrete type, converting the state where necessary.

This is done to further align the irqchip programming in the MSHV accelerator with the existing KVM code in QEMU. Subsequent commits will introduce AccelRouteChange to the MSHV accelerator code.

---
 accel/accel-irq.c           |  4 ++--
 accel/kvm/kvm-all.c         |  6 +++---
 accel/stubs/kvm-stub.c      |  2 +-
 hw/misc/ivshmem-pci.c       |  2 +-
 hw/vfio/pci.c               |  2 +-
 hw/virtio/virtio-pci.c      |  3 +--
 include/accel/accel-route.h | 17 +++++++++++++++++
 include/system/accel-irq.h  |  5 +++--
 include/system/kvm.h        | 21 ++++++++++-----------
 include/system/mshv.h       |  1 +
 target/i386/kvm/kvm.c       |  2 +-
 11 files changed, 41 insertions(+), 24 deletions(-)
 create mode 100644 include/accel/accel-route.h
accel/accel-irq.c +2 -2
index 7f864e35c4..0aa04c033d 100644
@@ -16,7 +16,7 @@
 #include "system/mshv.h"
 #include "system/accel-irq.h"
 
-int accel_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev)
+int accel_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
 #ifdef CONFIG_MSHV_IS_POSSIBLE
     if (mshv_msi_via_irqfd_enabled()) {
@@ -42,7 +42,7 @@ int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev)
     return -ENOSYS;
 }
 
-void accel_irqchip_commit_route_changes(KVMRouteChange *c)
+void accel_irqchip_commit_route_changes(AccelRouteChange *c)
 {
 #ifdef CONFIG_MSHV_IS_POSSIBLE
     if (mshv_msi_via_irqfd_enabled()) {
accel/kvm/kvm-all.c +3 -3
index 92af42503b..4e4234beff 100644
@@ -2359,11 +2359,11 @@ int kvm_irqchip_send_msi(KVMState *s, MSIMessage msg)
     return kvm_vm_ioctl(s, KVM_SIGNAL_MSI, &msi);
 }
 
-int kvm_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev)
+int kvm_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
     struct kvm_irq_routing_entry kroute = {};
     int virq;
-    KVMState *s = c->s;
+    KVMState *s = KVM_STATE(c->accel);
     MSIMessage msg = {0, 0};
 
     if (pci_available && dev) {
@@ -2506,7 +2506,7 @@ int kvm_irqchip_send_msi(KVMState *s, MSIMessage msg)
     abort();
 }
 
-int kvm_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev)
+int kvm_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
     return -ENOSYS;
 }
accel/stubs/kvm-stub.c +1 -1
index c4617caac6..32b4b07403 100644
@@ -44,7 +44,7 @@ int kvm_on_sigbus(int code, void *addr)
     return 1;
 }
 
-int kvm_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev)
+int kvm_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
     return -ENOSYS;
 }
hw/misc/ivshmem-pci.c +1 -1
index c987eebb98..aa8f271755 100644
@@ -424,7 +424,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector,
                                      Error **errp)
 {
     PCIDevice *pdev = PCI_DEVICE(s);
-    KVMRouteChange c;
+    AccelRouteChange c;
     int ret;
 
     IVSHMEM_DPRINTF("ivshmem_add_kvm_msi_virq vector:%d\n", vector);
hw/vfio/pci.c +1 -1
index 1945751ffd..79c503c8ae 100644
@@ -51,7 +51,7 @@
 #include "vfio-helpers.h"
 
 /* Protected by BQL */
-static KVMRouteChange vfio_route_change;
+static AccelRouteChange vfio_route_change;
 
 static void vfio_disable_interrupts(VFIOPCIDevice *vdev);
 static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled);
hw/virtio/virtio-pci.c +1 -2
index bcab2d18b8..5010572784 100644
@@ -869,7 +869,7 @@ static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,
     int ret;
 
     if (irqfd->users == 0) {
-        KVMRouteChange c = kvm_irqchip_begin_route_changes(kvm_state);
+        AccelRouteChange c = kvm_irqchip_begin_route_changes(kvm_state);
         ret = accel_irqchip_add_msi_route(&c, vector, &proxy->pci_dev);
         if (ret < 0) {
             return ret;
@@ -2695,4 +2695,3 @@ static void virtio_pci_register_types(void)
 }
 
 type_init(virtio_pci_register_types)
-
include/accel/accel-route.h new file +17 -0
new file mode 100644
index 0000000000..07fac27e2a
@@ -0,0 +1,17 @@
+/*
+ * Accelerator MSI route change tracking
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef ACCEL_ROUTE_H
+#define ACCEL_ROUTE_H
+
+#include "qemu/accel.h"
+
+typedef struct AccelRouteChange {
+    AccelState *accel;
+    int changes;
+} AccelRouteChange;
+
+#endif /* ACCEL_ROUTE_H */
include/system/accel-irq.h +3 -2
index a2caa06f54..a148920711 100644
@@ -25,9 +25,10 @@ static inline bool accel_irqchip_is_split(void)
     return mshv_msi_via_irqfd_enabled() || kvm_irqchip_is_split();
 }
 
-int accel_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev);
+int accel_irqchip_add_msi_route(AccelRouteChange *c, int vector,
+                                PCIDevice *dev);
 int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev);
-void accel_irqchip_commit_route_changes(KVMRouteChange *c);
+void accel_irqchip_commit_route_changes(AccelRouteChange *c);
 void accel_irqchip_commit_routes(void);
 void accel_irqchip_release_virq(int virq);
 int accel_irqchip_add_irqfd_notifier_gsi(EventNotifier *n, EventNotifier *rn,
include/system/kvm.h +10 -11
index 5fa33eddda..ccf90b8341 100644
@@ -18,6 +18,7 @@
 
 #include "exec/memattrs.h"
 #include "qemu/accel.h"
+#include "accel/accel-route.h"
 #include "qom/object.h"
 
 #ifdef COMPILING_PER_TARGET
@@ -183,11 +184,6 @@ extern KVMState *kvm_state;
 typedef struct Notifier Notifier;
 typedef struct NotifierWithReturn NotifierWithReturn;
 
-typedef struct KVMRouteChange {
-     KVMState *s;
-     int changes;
-} KVMRouteChange;
-
 /* external API */
 
 unsigned int kvm_get_max_memslots(void);
@@ -466,7 +462,7 @@ void kvm_init_cpu_signals(CPUState *cpu);
 
 /**
  * kvm_irqchip_add_msi_route - Add MSI route for specific vector
- * @c:      KVMRouteChange instance.
+ * @c:      AccelRouteChange instance.
  * @vector: which vector to add. This can be either MSI/MSIX
  *          vector. The function will automatically detect whether
  *          MSI/MSIX is enabled, and fetch corresponding MSI
@@ -475,20 +471,23 @@ void kvm_init_cpu_signals(CPUState *cpu);
  *          as @NULL, an empty MSI message will be inited.
  * @return: virq (>=0) when success, errno (<0) when failed.
  */
-int kvm_irqchip_add_msi_route(KVMRouteChange *c, int vector, PCIDevice *dev);
+int kvm_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev);
 int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg,
                                  PCIDevice *dev);
 void kvm_irqchip_commit_routes(KVMState *s);
 
-static inline KVMRouteChange kvm_irqchip_begin_route_changes(KVMState *s)
+static inline AccelRouteChange kvm_irqchip_begin_route_changes(KVMState *s)
 {
-    return (KVMRouteChange) { .s = s, .changes = 0 };
+    return (AccelRouteChange) {
+        .accel = ACCEL(s),
+        .changes = 0,
+    };
 }
 
-static inline void kvm_irqchip_commit_route_changes(KVMRouteChange *c)
+static inline void kvm_irqchip_commit_route_changes(AccelRouteChange *c)
 {
     if (c->changes) {
-        kvm_irqchip_commit_routes(c->s);
+        kvm_irqchip_commit_routes(KVM_STATE(c->accel));
         c->changes = 0;
     }
 }
include/system/mshv.h +1 -0
index 75286baf16..1e96b3a606 100644
@@ -21,6 +21,7 @@
 #include "qapi/qapi-types-common.h"
 #include "system/memory.h"
 #include "accel/accel-ops.h"
+#include "accel/accel-route.h"
 
 #ifdef COMPILING_PER_TARGET
 #ifdef CONFIG_MSHV
target/i386/kvm/kvm.c +1 -1
index 9e352882c8..478a5573cd 100644
@@ -6678,7 +6678,7 @@ void kvm_arch_init_irq_routing(KVMState *s)
     kvm_gsi_routing_allowed = true;
 
     if (kvm_irqchip_is_split()) {
-        KVMRouteChange c = kvm_irqchip_begin_route_changes(s);
+        AccelRouteChange c = kvm_irqchip_begin_route_changes(s);
         int i;
 
         /* If the ioapic is in QEMU and the lapics are in KVM, reserve
Signature · 1 line
2.34.1
#5 lore raw

PATCH 5/34 accel/accel-irq: add generic begin_route_changes

In reply to Magnus Kulke

A generic accel_irqchip_begin_route_change() fn has been introduced for usage in the MSHV accelerator. It replaces the respective kvm_ fn.

---
 accel/accel-irq.c          | 21 +++++++++++++++++++++
 hw/misc/ivshmem-pci.c      |  4 ++--
 hw/vfio/pci.c              |  5 +++--
 hw/virtio/virtio-pci.c     |  2 +-
 include/system/accel-irq.h |  1 +
 include/system/kvm.h       |  8 --------
 target/i386/kvm/kvm.c      |  3 ++-
 7 files changed, 30 insertions(+), 14 deletions(-)
accel/accel-irq.c +21 -0
index 0aa04c033d..3815f6727c 100644
@@ -10,6 +10,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qemu/error-report.h"
 #include "hw/pci/msi.h"
 
 #include "system/kvm.h"
@@ -104,3 +105,23 @@ int accel_irqchip_remove_irqfd_notifier_gsi(EventNotifier *n, int virq)
     }
     return -ENOSYS;
 }
+
+inline AccelRouteChange accel_irqchip_begin_route_changes(void)
+{
+#ifdef CONFIG_MSHV_IS_POSSIBLE
+    if (mshv_msi_via_irqfd_enabled()) {
+        return (AccelRouteChange) {
+            .accel = ACCEL(mshv_state),
+            .changes = 0,
+        };
+    }
+#endif
+    if (kvm_enabled()) {
+        return (AccelRouteChange) {
+            .accel = ACCEL(kvm_state),
+            .changes = 0,
+        };
+    }
+    error_report("can't initiate route change, no accel irqchip available");
+    abort();
+}
hw/misc/ivshmem-pci.c +2 -2
index aa8f271755..fa23562886 100644
@@ -26,7 +26,7 @@
 #include "hw/core/qdev-properties-system.h"
 #include "hw/pci/msi.h"
 #include "hw/pci/msix.h"
-#include "system/kvm.h"
+#include "system/accel-irq.h"
 #include "migration/blocker.h"
 #include "migration/vmstate.h"
 #include "qemu/error-report.h"
@@ -430,7 +430,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector,
     IVSHMEM_DPRINTF("ivshmem_add_kvm_msi_virq vector:%d\n", vector);
     assert(!s->msi_vectors[vector].pdev);
 
-    c = kvm_irqchip_begin_route_changes(kvm_state);
+    c = accel_irqchip_begin_route_changes();
     ret = kvm_irqchip_add_msi_route(&c, vector, pdev);
     if (ret < 0) {
         error_setg(errp, "kvm_irqchip_add_msi_route failed");
hw/vfio/pci.c +3 -2
index 79c503c8ae..35b4545482 100644
@@ -39,6 +39,7 @@
 #include "qemu/module.h"
 #include "qemu/range.h"
 #include "qemu/units.h"
+#include "system/accel-irq.h"
 #include "system/kvm.h"
 #include "system/runstate.h"
 #include "pci.h"
@@ -692,7 +693,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,
             if (vdev->defer_kvm_irq_routing) {
                 vfio_pci_add_kvm_msi_virq(vdev, vector, nr, true);
             } else {
-                vfio_route_change = kvm_irqchip_begin_route_changes(kvm_state);
+                vfio_route_change = accel_irqchip_begin_route_changes();
                 vfio_pci_add_kvm_msi_virq(vdev, vector, nr, true);
                 kvm_irqchip_commit_route_changes(&vfio_route_change);
                 vfio_connect_kvm_msi_virq(vector, nr);
@@ -793,7 +794,7 @@ void vfio_pci_prepare_kvm_msi_virq_batch(VFIOPCIDevice *vdev)
 {
     assert(!vdev->defer_kvm_irq_routing);
     vdev->defer_kvm_irq_routing = true;
-    vfio_route_change = kvm_irqchip_begin_route_changes(kvm_state);
+    vfio_route_change = accel_irqchip_begin_route_changes();
 }
 
 void vfio_pci_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev)
hw/virtio/virtio-pci.c +1 -1
index 5010572784..faa4a41cca 100644
@@ -869,7 +869,7 @@ static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,
     int ret;
 
     if (irqfd->users == 0) {
-        AccelRouteChange c = kvm_irqchip_begin_route_changes(kvm_state);
+        AccelRouteChange c = accel_irqchip_begin_route_changes();
         ret = accel_irqchip_add_msi_route(&c, vector, &proxy->pci_dev);
         if (ret < 0) {
             return ret;
include/system/accel-irq.h +1 -0
index a148920711..fc94c54264 100644
@@ -25,6 +25,7 @@ static inline bool accel_irqchip_is_split(void)
     return mshv_msi_via_irqfd_enabled() || kvm_irqchip_is_split();
 }
 
+AccelRouteChange accel_irqchip_begin_route_changes(void);
 int accel_irqchip_add_msi_route(AccelRouteChange *c, int vector,
                                 PCIDevice *dev);
 int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev);
include/system/kvm.h +0 -8
index ccf90b8341..fec24d2135 100644
@@ -476,14 +476,6 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg,
                                  PCIDevice *dev);
 void kvm_irqchip_commit_routes(KVMState *s);
 
-static inline AccelRouteChange kvm_irqchip_begin_route_changes(KVMState *s)
-{
-    return (AccelRouteChange) {
-        .accel = ACCEL(s),
-        .changes = 0,
-    };
-}
-
 static inline void kvm_irqchip_commit_route_changes(AccelRouteChange *c)
 {
     if (c->changes) {
target/i386/kvm/kvm.c +2 -1
index 478a5573cd..03312922d5 100644
@@ -32,6 +32,7 @@
 #include "vmsr_energy.h"
 #include "system/system.h"
 #include "system/hw_accel.h"
+#include "system/accel-irq.h"
 #include "system/kvm_int.h"
 #include "system/runstate.h"
 #include "system/ramblock.h"
@@ -6678,7 +6679,7 @@ void kvm_arch_init_irq_routing(KVMState *s)
     kvm_gsi_routing_allowed = true;
 
     if (kvm_irqchip_is_split()) {
-        AccelRouteChange c = kvm_irqchip_begin_route_changes(s);
+        AccelRouteChange c = accel_irqchip_begin_route_changes();
         int i;
 
         /* If the ioapic is in QEMU and the lapics are in KVM, reserve
Signature · 1 line
2.34.1
#6 lore raw

PATCH 6/34 accel/accel-irq: add generic commit_route_changes

In reply to Magnus Kulke

A generic accel_irqchip_commit_route_changes() fn has been introduced for usage in the MSHV accelerator. The respective kvm_ fn can be removed since we handle the commit op in a generic way.

---
 accel/accel-irq.c     | 10 +++-------
 hw/misc/ivshmem-pci.c |  2 +-
 hw/vfio/pci.c         |  4 ++--
 include/system/kvm.h  |  8 --------
 target/i386/kvm/kvm.c |  2 +-
 5 files changed, 7 insertions(+), 19 deletions(-)
accel/accel-irq.c +3 -7
index 3815f6727c..7e71b52555 100644
@@ -45,13 +45,9 @@ int accel_irqchip_update_msi_route(int vector, MSIMessage msg, PCIDevice *dev)
 
 void accel_irqchip_commit_route_changes(AccelRouteChange *c)
 {
-#ifdef CONFIG_MSHV_IS_POSSIBLE
-    if (mshv_msi_via_irqfd_enabled()) {
-        mshv_irqchip_commit_routes();
-    }
-#endif
-    if (kvm_enabled()) {
-        kvm_irqchip_commit_route_changes(c);
+    if (c->changes) {
+        accel_irqchip_commit_routes();
+        c->changes = 0;
     }
 }
 
hw/misc/ivshmem-pci.c +1 -1
index fa23562886..536475e9de 100644
@@ -436,7 +436,7 @@ static void ivshmem_add_kvm_msi_virq(IVShmemState *s, int vector,
         error_setg(errp, "kvm_irqchip_add_msi_route failed");
         return;
     }
-    kvm_irqchip_commit_route_changes(&c);
+    accel_irqchip_commit_route_changes(&c);
 
     s->msi_vectors[vector].virq = ret;
     s->msi_vectors[vector].pdev = pdev;
hw/vfio/pci.c +2 -2
index 35b4545482..fb20902412 100644
@@ -695,7 +695,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr,
             } else {
                 vfio_route_change = accel_irqchip_begin_route_changes();
                 vfio_pci_add_kvm_msi_virq(vdev, vector, nr, true);
-                kvm_irqchip_commit_route_changes(&vfio_route_change);
+                accel_irqchip_commit_route_changes(&vfio_route_change);
                 vfio_connect_kvm_msi_virq(vector, nr);
             }
         }
@@ -804,7 +804,7 @@ void vfio_pci_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev)
     assert(vdev->defer_kvm_irq_routing);
     vdev->defer_kvm_irq_routing = false;
 
-    kvm_irqchip_commit_route_changes(&vfio_route_change);
+    accel_irqchip_commit_route_changes(&vfio_route_change);
 
     for (i = 0; i < vdev->nr_vectors; i++) {
         vfio_connect_kvm_msi_virq(&vdev->msi_vectors[i], i);
include/system/kvm.h +0 -8
index fec24d2135..cdd1856ac5 100644
@@ -476,14 +476,6 @@ int kvm_irqchip_update_msi_route(KVMState *s, int virq, MSIMessage msg,
                                  PCIDevice *dev);
 void kvm_irqchip_commit_routes(KVMState *s);
 
-static inline void kvm_irqchip_commit_route_changes(AccelRouteChange *c)
-{
-    if (c->changes) {
-        kvm_irqchip_commit_routes(KVM_STATE(c->accel));
-        c->changes = 0;
-    }
-}
-
 int kvm_irqchip_get_virq(KVMState *s);
 void kvm_irqchip_release_virq(KVMState *s, int virq);
 
target/i386/kvm/kvm.c +1 -1
index 03312922d5..64cc421abe 100644
@@ -6690,7 +6690,7 @@ void kvm_arch_init_irq_routing(KVMState *s)
                 exit(1);
             }
         }
-        kvm_irqchip_commit_route_changes(&c);
+        accel_irqchip_commit_route_changes(&c);
     }
 }
 
Signature · 1 line
2.34.1
#7 lore raw

PATCH 7/34 accel/mshv: add irq_routes to state

In reply to Magnus Kulke

This change adds fields related to irq routing to the MSHV state, following similar fields in the KVM implementation.

So far the fields are only initialized, they will be used in subsequent commits for bookkeeping purposes and storing uncommitted interrupt routes.

The TYPE_MSHV_ACCEL defines have been moved to the header.

---
 accel/mshv/irq.c          | 10 ++++++++++
 accel/mshv/mshv-all.c     |  6 ++----
 include/system/mshv.h     |  7 +++++++
 include/system/mshv_int.h |  5 +++++
 4 files changed, 24 insertions(+), 4 deletions(-)
accel/mshv/irq.c +10 -0
index 3c238c33c3..82f2022c7c 100644
@@ -396,3 +396,13 @@ int mshv_reserve_ioapic_msi_routes(int vm_fd)
 
     return 0;
 }
+
+void mshv_init_irq_routing(MshvState *s)
+{
+    int gsi_count = MSHV_MAX_MSI_ROUTES;
+
+    s->irq_routes = g_malloc0(sizeof(*s->irq_routes));
+    s->nr_allocated_irq_routes = 0;
+    s->gsi_count = gsi_count;
+    s->used_gsi_bitmap = bitmap_new(gsi_count);
+}
accel/mshv/mshv-all.c +2 -4
index 04d248fe1d..8acb080db1 100644
@@ -43,10 +43,6 @@
 #include <err.h>
 #include <sys/ioctl.h>
 
-#define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv")
-
-DECLARE_INSTANCE_CHECKER(MshvState, MSHV_STATE, TYPE_MSHV_ACCEL)
-
 bool mshv_allowed;
 
 MshvState *mshv_state;
@@ -457,6 +453,8 @@ static int mshv_init(AccelState *as, MachineState *ms)
 
     mshv_state = s;
 
+    mshv_init_irq_routing(s);
+
     register_mshv_memory_listener(s, &s->memory_listener, &address_space_memory,
                                   0, "mshv-memory");
     memory_listener_register(&mshv_io_listener, &address_space_io);
include/system/mshv.h +7 -0
index 1e96b3a606..0d1745315b 100644
@@ -45,7 +45,13 @@ extern bool mshv_allowed;
 #define mshv_msi_via_irqfd_enabled() mshv_enabled()
 #endif
 
+#define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv")
+
 typedef struct MshvState MshvState;
+
+DECLARE_INSTANCE_CHECKER(MshvState, MSHV_STATE,
+                         TYPE_MSHV_ACCEL)
+
 extern MshvState *mshv_state;
 
 /* interrupt */
@@ -60,5 +66,6 @@ void mshv_irqchip_release_virq(int virq);
 int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *n,
                                         const EventNotifier *rn, int virq);
 int mshv_irqchip_remove_irqfd_notifier_gsi(const EventNotifier *n, int virq);
+void mshv_init_irq_routing(MshvState *s);
 
 #endif
include/system/mshv_int.h +5 -0
index 70631ca6ba..56fda76a9c 100644
@@ -48,6 +48,11 @@ struct MshvState {
     int nr_as;
     MshvAddressSpace *as;
     int fd;
+    /* irqchip routing */
+    struct mshv_user_irq_table *irq_routes;
+    int nr_allocated_irq_routes;
+    unsigned long *used_gsi_bitmap;
+    unsigned int gsi_count;
 };
 
 typedef struct MshvMsiControl {
Signature · 1 line
2.34.1
#8 lore raw

PATCH 8/34 accel/mshv: update s->irq_routes in add_msi_route

In reply to Magnus Kulke

The irq_routes field of the state is populated with native mshv irq route entries. The allocation logic is modelled after the KVM
implementation: we will always allocate a minumum of 64 entries and use a bitmask to find/set/clear GSIs.

The old implementation of add_msi_routes will be removed in a subsequent commit.

---
 accel/accel-irq.c       |  2 +-
 accel/mshv/irq.c        | 87 +++++++++++++++++++++++++++++++++++++----
 accel/stubs/mshv-stub.c |  2 +-
 include/system/mshv.h   |  3 +-
 4 files changed, 84 insertions(+), 10 deletions(-)
accel/accel-irq.c +1 -1
index 7e71b52555..5a97a345b2 100644
@@ -21,7 +21,7 @@ int accel_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
 #ifdef CONFIG_MSHV_IS_POSSIBLE
     if (mshv_msi_via_irqfd_enabled()) {
-        return mshv_irqchip_add_msi_route(vector, dev);
+        return mshv_irqchip_add_msi_route(c, vector, dev);
     }
 #endif
     if (kvm_enabled()) {
accel/mshv/irq.c +80 -7

98 lines not on this page · raw message

accel/stubs/mshv-stub.c +1 -1
index e499b199d9..998c9e2fc6 100644
@@ -14,7 +14,7 @@
 
 bool mshv_allowed;
 
-int mshv_irqchip_add_msi_route(int vector, PCIDevice *dev)
+int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
 {
     return -ENOSYS;
 }
include/system/mshv.h +2 -1
index 0d1745315b..7f60aba308 100644
@@ -33,6 +33,7 @@
 #endif
 
 #define MSHV_MAX_MSI_ROUTES 4096
+#define MSHV_MIN_ALLOCATED_MSI_ROUTES 64
 
 #define MSHV_PAGE_SHIFT 12
 
@@ -59,7 +60,7 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3
                            uint32_t vp_index, bool logical_destination_mode,
                            bool level_triggered);
 
-int mshv_irqchip_add_msi_route(int vector, PCIDevice *dev);
+int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev);
 int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev);
 void mshv_irqchip_commit_routes(void);
 void mshv_irqchip_release_virq(int virq);
Signature · 1 line
2.34.1
#9 lore raw

PATCH 9/34 accel/mshv: update s->irq_routes in update_msi_route

In reply to Magnus Kulke

The state's irq_routes field will be updated when an irqchip's gsi is requested to be updated with a new dest/vector.

The old set_msi_routing() fn is redundant and can be removed.

---
 accel/mshv/irq.c | 87 +++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 49 deletions(-)
accel/mshv/irq.c +38 -49
index 9d6bdde27a..b5a047b367 100644
@@ -36,52 +36,6 @@ void mshv_init_msicontrol(void)
     msi_control->updated = false;
 }
 
-static int set_msi_routing(uint32_t gsi, uint64_t addr, uint32_t data)
-{
-    struct mshv_user_irq_entry *entry;
-    uint32_t high_addr = addr >> 32;
-    uint32_t low_addr = addr & 0xFFFFFFFF;
-    GHashTable *gsi_routes;
-
-    trace_mshv_set_msi_routing(gsi, addr, data);
-
-    if (gsi >= MSHV_MAX_MSI_ROUTES) {
-        error_report("gsi >= MSHV_MAX_MSI_ROUTES");
-        return -1;
-    }
-
-    assert(msi_control);
-
-    WITH_QEMU_LOCK_GUARD(&msi_control_mutex) {
-        gsi_routes = msi_control->gsi_routes;
-        entry = g_hash_table_lookup(gsi_routes, GINT_TO_POINTER(gsi));
-
-        if (entry
-            && entry->address_hi == high_addr
-            && entry->address_lo == low_addr
-            && entry->data == data)
-        {
-            /* nothing to update */
-            return 0;
-        }
-
-        /* free old entry */
-        g_free(entry);
-
-        /* create new entry */
-        entry = g_new0(struct mshv_user_irq_entry, 1);
-        entry->gsi = gsi;
-        entry->address_hi = high_addr;
-        entry->address_lo = low_addr;
-        entry->data = data;
-
-        g_hash_table_insert(gsi_routes, GINT_TO_POINTER(gsi), entry);
-        msi_control->updated = true;
-    }
-
-    return 0;
-}
-
 static int add_msi_routing(uint64_t addr, uint32_t data)
 {
     struct mshv_user_irq_entry *route_entry;
@@ -370,16 +324,51 @@ void mshv_irqchip_release_virq(int virq)
     remove_msi_routing(virq);
 }
 
+static int update_routing_entry(MshvState *s,
+                                struct mshv_user_irq_entry *new_entry)
+{
+    struct mshv_user_irq_entry *entry;
+    int n;
+
+    for (n = 0; n < s->irq_routes->nr; n++) {
+        entry = &s->irq_routes->entries[n];
+        if (entry->gsi != new_entry->gsi) {
+            continue;
+        }
+
+        if (!memcmp(entry, new_entry, sizeof *entry)) {
+            return 0;
+        }
+
+        *entry = *new_entry;
+
+        return 0;
+    }
+
+    return -ESRCH;
+}
+
 int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev)
 {
+    uint32_t addr_hi = msg.address >> 32;
+    uint32_t addr_lo = msg.address & 0xFFFFFFFF;
+    uint32_t data = le32_to_cpu(msg.data);
+    struct mshv_user_irq_entry entry = {
+        .gsi = virq,
+        .address_hi = addr_hi,
+        .address_lo = addr_lo,
+        .data = data,
+    };
     int ret;
 
-    ret = set_msi_routing(virq, msg.address, le32_to_cpu(msg.data));
+    ret = update_routing_entry(mshv_state, &entry);
     if (ret < 0) {
-        error_report("Failed to set msi routing");
-        return -1;
+        error_report("Failed to set msi routing for gsi %d", virq);
+        abort();
     }
 
+    trace_mshv_set_msi_routing(virq, msg.address, data);
+
     return 0;
 }
 
Signature · 1 line
2.34.1
#10 lore raw

PATCH 10/34 accel/mshv: update s->irq_routes in release_virq

In reply to Magnus Kulke

The state's irq_routes field will be updated when an irqchip's gsi is requested to be released.

The old remove_msi_routing() fn is redundant and can be removed.

---
 accel/accel-irq.c       |  2 +-
 accel/mshv/irq.c        | 43 ++++++++++++++---------------------------
 accel/stubs/mshv-stub.c |  2 +-
 include/system/mshv.h   |  2 +-
 4 files changed, 17 insertions(+), 32 deletions(-)
accel/accel-irq.c +1 -1
index 5a97a345b2..0cb3ef78d3 100644
@@ -67,7 +67,7 @@ void accel_irqchip_release_virq(int virq)
 {
 #ifdef CONFIG_MSHV_IS_POSSIBLE
     if (mshv_msi_via_irqfd_enabled()) {
-        mshv_irqchip_release_virq(virq);
+        mshv_irqchip_release_virq(mshv_state, virq);
     }
 #endif
     if (kvm_enabled()) {
accel/mshv/irq.c +14 -29

56 lines not on this page · raw message

accel/stubs/mshv-stub.c +1 -1
index 998c9e2fc6..dadf05511a 100644
@@ -19,7 +19,7 @@ int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev)
     return -ENOSYS;
 }
 
-void mshv_irqchip_release_virq(int virq)
+void mshv_irqchip_release_virq(MshvState *s, int virq)
 {
 }
 
include/system/mshv.h +1 -1
index 7f60aba308..2033beed70 100644
@@ -63,7 +63,7 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3
 int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev);
 int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev);
 void mshv_irqchip_commit_routes(void);
-void mshv_irqchip_release_virq(int virq);
+void mshv_irqchip_release_virq(MshvState *s, int virq);
 int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *n,
                                         const EventNotifier *rn, int virq);
 int mshv_irqchip_remove_irqfd_notifier_gsi(const EventNotifier *n, int virq);
Signature · 1 line
2.34.1
#11 lore raw

PATCH 11/34 accel/mshv: use s->irq_routes in commit_routes

In reply to Magnus Kulke

In mshv_irqchip_commit_routes() the entries that have been accumulated in s->irq_routes are committed directly to MSHV's irqchip.

The old commit_msi_routing_table() fn will be removed in a subsquent commit.

---
 accel/accel-irq.c       | 2 +-
 accel/mshv/irq.c        | 7 ++++---
 accel/stubs/mshv-stub.c | 2 +-
 include/system/mshv.h   | 2 +-
 4 files changed, 7 insertions(+), 6 deletions(-)
accel/accel-irq.c +1 -1
index 0cb3ef78d3..4303e10e40 100644
@@ -55,7 +55,7 @@ void accel_irqchip_commit_routes(void)
 {
 #ifdef CONFIG_MSHV_IS_POSSIBLE
     if (mshv_msi_via_irqfd_enabled()) {
-        mshv_irqchip_commit_routes();
+        mshv_irqchip_commit_routes(mshv_state);
     }
 #endif
     if (kvm_enabled()) {
accel/mshv/irq.c +4 -3
index 990ce34620..9ba837f0e2 100644
@@ -394,16 +394,17 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3
     return 0;
 }
 
-void mshv_irqchip_commit_routes(void)
+void mshv_irqchip_commit_routes(MshvState *s)
 {
     int ret;
-    int vm_fd = mshv_state->vm;
+    int vm_fd = s->vm;
 
-    ret = commit_msi_routing_table(vm_fd);
+    ret = ioctl(vm_fd, MSHV_SET_MSI_ROUTING, s->irq_routes);
     if (ret < 0) {
         error_report("Failed to commit msi routing table");
         abort();
     }
+    trace_mshv_commit_msi_routing_table(vm_fd, s->irq_routes->nr);
 }
 
 int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *event,
accel/stubs/mshv-stub.c +1 -1
index dadf05511a..8b69539a85 100644
@@ -28,7 +28,7 @@ int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev)
     return -ENOSYS;
 }
 
-void mshv_irqchip_commit_routes(void)
+void mshv_irqchip_commit_routes(MshvState *s)
 {
 }
 
include/system/mshv.h +1 -1
index 2033beed70..0c9a4f927c 100644
@@ -62,8 +62,8 @@ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint3
 
 int mshv_irqchip_add_msi_route(AccelRouteChange *c, int vector, PCIDevice *dev);
 int mshv_irqchip_update_msi_route(int virq, MSIMessage msg, PCIDevice *dev);
-void mshv_irqchip_commit_routes(void);
 void mshv_irqchip_release_virq(MshvState *s, int virq);
+void mshv_irqchip_commit_routes(MshvState *s);
 int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *n,
                                         const EventNotifier *rn, int virq);
 int mshv_irqchip_remove_irqfd_notifier_gsi(const EventNotifier *n, int virq);
Signature · 1 line
2.34.1
#12 lore raw

PATCH 12/34 accel/mshv: reserve ioapic routes on s->irq_routes

In reply to Magnus Kulke

We reserve 24 ioapic routes using the new functions that operate on the mshv apic state.

commit/add_msi_routing() fn's can be removed now.

---
 accel/mshv/irq.c          | 115 ++++++--------------------------------
 accel/mshv/mshv-all.c     |   5 --
 include/system/mshv_int.h |   1 -
 3 files changed, 18 insertions(+), 103 deletions(-)
accel/mshv/irq.c +18 -97
index 9ba837f0e2..52b8ac9479 100644
@@ -36,93 +36,6 @@ void mshv_init_msicontrol(void)
     msi_control->updated = false;
 }
 
-static int add_msi_routing(uint64_t addr, uint32_t data)
-{
-    struct mshv_user_irq_entry *route_entry;
-    uint32_t high_addr = addr >> 32;
-    uint32_t low_addr = addr & 0xFFFFFFFF;
-    int gsi;
-    GHashTable *gsi_routes;
-
-    trace_mshv_add_msi_routing(addr, data);
-
-    assert(msi_control);
-
-    WITH_QEMU_LOCK_GUARD(&msi_control_mutex) {
-        /* find an empty slot */
-        gsi = 0;
-        gsi_routes = msi_control->gsi_routes;
-        while (gsi < MSHV_MAX_MSI_ROUTES) {
-            route_entry = g_hash_table_lookup(gsi_routes, GINT_TO_POINTER(gsi));
-            if (!route_entry) {
-                break;
-            }
-            gsi++;
-        }
-        if (gsi >= MSHV_MAX_MSI_ROUTES) {
-            error_report("No empty gsi slot available");
-            return -1;
-        }
-
-        /* create new entry */
-        route_entry = g_new0(struct mshv_user_irq_entry, 1);
-        route_entry->gsi = gsi;
-        route_entry->address_hi = high_addr;
-        route_entry->address_lo = low_addr;
-        route_entry->data = data;
-
-        g_hash_table_insert(gsi_routes, GINT_TO_POINTER(gsi), route_entry);
-        msi_control->updated = true;
-    }
-
-    return gsi;
-}
-
-static int commit_msi_routing_table(int vm_fd)
-{
-    guint len;
-    int i, ret;
-    size_t table_size;
-    struct mshv_user_irq_table *table;
-    GHashTableIter iter;
-    gpointer key, value;
-
-    assert(msi_control);
-
-    WITH_QEMU_LOCK_GUARD(&msi_control_mutex) {
-        if (!msi_control->updated) {
-            /* nothing to update */
-            return 0;
-        }
-
-        /* Calculate the size of the table */
-        len = g_hash_table_size(msi_control->gsi_routes);
-        table_size = sizeof(struct mshv_user_irq_table)
-                     + len * sizeof(struct mshv_user_irq_entry);
-        table = g_malloc0(table_size);
-
-        g_hash_table_iter_init(&iter, msi_control->gsi_routes);
-        i = 0;
-        while (g_hash_table_iter_next(&iter, &key, &value)) {
-            struct mshv_user_irq_entry *entry = value;
-            table->entries[i] = *entry;
-            i++;
-        }
-        table->nr = i;
-
-        trace_mshv_commit_msi_routing_table(vm_fd, len);
-
-        ret = ioctl(vm_fd, MSHV_SET_MSI_ROUTING, table);
-        g_free(table);
-        if (ret < 0) {
-            error_report("Failed to commit msi routing table");
-            return -1;
-        }
-        msi_control->updated = false;
-    }
-    return 0;
-}
-
 /* Pass an eventfd which is to be used for injecting interrupts from userland */
 static int irqfd(int vm_fd, int fd, int resample_fd, uint32_t gsi,
                  uint32_t flags)
@@ -420,37 +333,45 @@ int mshv_irqchip_remove_irqfd_notifier_gsi(const EventNotifier *event,
     return irqchip_update_irqfd_notifier_gsi(event, NULL, virq, false);
 }
 
-int mshv_reserve_ioapic_msi_routes(int vm_fd)
+static int mshv_reserve_ioapic_msi_routes(MshvState *s)
 {
-    int ret, gsi;
+    int ret, i;
+    int gsi = 0;
+    struct mshv_user_irq_entry blank_entry = { 0 };
 
     /*
      * Reserve GSI 0-23 for IOAPIC pins, to avoid conflicts of legacy
      * peripherals with MSI-X devices
      */
-    for (gsi = 0; gsi < IOAPIC_NUM_PINS; gsi++) {
-        ret = add_msi_routing(0, 0);
+    for (i = 0; i < IOAPIC_NUM_PINS; i++) {
+        /* ret = add_msi_routing(0, 0); */
+        ret = irqchip_allocate_gsi(s, &gsi);
         if (ret < 0) {
-            error_report("Failed to reserve GSI %d", gsi);
+            error_report("Failed to reserve GSI %d: %s", gsi, strerror(-ret));
             return -1;
         }
+        blank_entry.gsi = gsi;
+        add_routing_entry(s, &blank_entry);
     }
 
-    ret = commit_msi_routing_table(vm_fd);
-    if (ret < 0) {
-        error_report("Failed to commit reserved IOAPIC MSI routes");
-        return -1;
-    }
+    mshv_irqchip_commit_routes(s);
 
     return 0;
 }
 
 void mshv_init_irq_routing(MshvState *s)
 {
+    int ret;
     int gsi_count = MSHV_MAX_MSI_ROUTES;
 
     s->irq_routes = g_malloc0(sizeof(*s->irq_routes));
     s->nr_allocated_irq_routes = 0;
     s->gsi_count = gsi_count;
     s->used_gsi_bitmap = bitmap_new(gsi_count);
+
+    ret = mshv_reserve_ioapic_msi_routes(s);
+    if (ret < 0) {
+        error_report("Failed to reserve IOAPIC MSI routes");
+        abort();
+    }
 }
accel/mshv/mshv-all.c +0 -5
index 8acb080db1..08bc26713f 100644
@@ -200,11 +200,6 @@ static int create_vm(int mshv_fd, int *vm_fd)
         return -1;
     }
 
-    ret = mshv_reserve_ioapic_msi_routes(*vm_fd);
-    if (ret < 0) {
-        return -1;
-    }
-
     ret = mshv_arch_post_init_vm(*vm_fd);
     if (ret < 0) {
         return -1;
include/system/mshv_int.h +0 -1
index 56fda76a9c..9bc56e70cf 100644
@@ -120,6 +120,5 @@ int mshv_configure_msr(const CPUState *cpu, const MshvMsrEntry *msrs,
 
 /* interrupt */
 void mshv_init_msicontrol(void);
-int mshv_reserve_ioapic_msi_routes(int vm_fd);
 
 #endif
Signature · 1 line
2.34.1
#13 lore raw

PATCH 13/34 accel/mshv: remove redundant msi controller

In reply to Magnus Kulke

The remaining MsiControl infrastructure can be removed now

---
 accel/mshv/irq.c          | 11 -----------
 accel/mshv/mshv-all.c     |  2 --
 include/system/mshv_int.h |  3 ---
 3 files changed, 16 deletions(-)
accel/mshv/irq.c +0 -11
index 52b8ac9479..4828ac51ac 100644
@@ -25,17 +25,6 @@
 #define MSHV_IRQFD_RESAMPLE_FLAG (1 << MSHV_IRQFD_BIT_RESAMPLE)
 #define MSHV_IRQFD_BIT_DEASSIGN_FLAG (1 << MSHV_IRQFD_BIT_DEASSIGN)
 
-static MshvMsiControl *msi_control;
-static QemuMutex msi_control_mutex;
-
-void mshv_init_msicontrol(void)
-{
-    qemu_mutex_init(&msi_control_mutex);
-    msi_control = g_new0(MshvMsiControl, 1);
-    msi_control->gsi_routes = g_hash_table_new(g_direct_hash, g_direct_equal);
-    msi_control->updated = false;
-}
-
 /* Pass an eventfd which is to be used for injecting interrupts from userland */
 static int irqfd(int vm_fd, int fd, int resample_fd, uint32_t gsi,
                  uint32_t flags)
accel/mshv/mshv-all.c +0 -2
index 08bc26713f..056b19b3b8 100644
@@ -426,8 +426,6 @@ static int mshv_init(AccelState *as, MachineState *ms)
 
     mshv_init_mmio_emu();
 
-    mshv_init_msicontrol();
-
     ret = create_vm(mshv_fd, &vm_fd);
     if (ret < 0) {
         close(mshv_fd);
include/system/mshv_int.h +0 -3
index 9bc56e70cf..7af5bcf022 100644
@@ -118,7 +118,4 @@ typedef struct MshvMsrEntries {
 int mshv_configure_msr(const CPUState *cpu, const MshvMsrEntry *msrs,
                        size_t n_msrs);
 
-/* interrupt */
-void mshv_init_msicontrol(void);
-
 #endif
Signature · 1 line
2.34.1
#14 lore raw

PATCH 14/34 target/i386/mshv: move apic logic into own file

In reply to Magnus Kulke

Use a new file to unclutter the vcpu code gradually from APIC related logic.

---
 include/system/mshv_int.h    |  8 ++++
 target/i386/mshv/meson.build |  1 +
 target/i386/mshv/mshv-apic.c | 78 ++++++++++++++++++++++++++++++++++++
 3 files changed, 87 insertions(+)
 create mode 100644 target/i386/mshv/mshv-apic.c
include/system/mshv_int.h +8 -0
index 7af5bcf022..f86c7a3be6 100644
@@ -16,6 +16,8 @@
 
 #define MSHV_MSR_ENTRIES_COUNT 64
 
+struct mshv_get_set_vp_state;
+
 typedef struct hyperv_message hv_message;
 
 typedef struct MshvHvCallArgs {
@@ -84,6 +86,12 @@ void mshv_arch_destroy_vcpu(CPUState *cpu);
 void mshv_arch_amend_proc_features(
     union hv_partition_synthetic_processor_features *features);
 int mshv_arch_post_init_vm(int vm_fd);
+int mshv_set_lapic(int cpu_fd,
+                   const struct hv_local_interrupt_controller_state *state);
+int mshv_get_lapic(int cpu_fd,
+                   struct hv_local_interrupt_controller_state *state);
+int mshv_get_vp_state(int cpu_fd, struct mshv_get_set_vp_state *state);
+int mshv_set_vp_state(int cpu_fd, const struct mshv_get_set_vp_state *state);
 
 typedef struct mshv_root_hvcall mshv_root_hvcall;
 int mshv_hvcall(int fd, const mshv_root_hvcall *args);
target/i386/mshv/meson.build +1 -0
index 49f28d4a5b..5eb6e833a6 100644
@@ -1,6 +1,7 @@
 i386_mshv_ss = ss.source_set()
 
 i386_mshv_ss.add(files(
+  'mshv-apic.c',
   'mshv-cpu.c',
 ))
 
target/i386/mshv/mshv-apic.c new file +78 -0

78 lines not on this page · raw message

Signature · 1 line
2.34.1
#15 lore raw

PATCH 15/34 target/i386/mshv: remove redundant apic helpers

In reply to Magnus Kulke

This change remove the now redundant apic helpers in mshv-cpu.c

---
 target/i386/mshv/mshv-cpu.c | 88 ++-----------------------------------
 1 file changed, 3 insertions(+), 85 deletions(-)
target/i386/mshv/mshv-cpu.c +3 -85
index 56656ac0b0..12343acb01 100644
@@ -919,101 +919,19 @@ static int set_xc_reg(const CPUState *cpu)
     return 0;
 }
 
-static int get_vp_state(int cpu_fd, struct mshv_get_set_vp_state *state)
-{
-    int ret;
-
-    ret = ioctl(cpu_fd, MSHV_GET_VP_STATE, state);
-    if (ret < 0) {
-        error_report("failed to get partition state: %s", strerror(errno));
-        return -1;
-    }
-
-    return 0;
-}
-
-static int get_lapic(const CPUState *cpu,
-                     struct hv_local_interrupt_controller_state *state)
-{
-    int ret;
-    size_t size = 4096;
-    /* buffer aligned to 4k, as *state requires that */
-    void *buffer = qemu_memalign(size, size);
-    struct mshv_get_set_vp_state mshv_state = { 0 };
-    int cpu_fd = mshv_vcpufd(cpu);
-
-    mshv_state.buf_ptr = (uint64_t) buffer;
-    mshv_state.buf_sz = size;
-    mshv_state.type = MSHV_VP_STATE_LAPIC;
-
-    ret = get_vp_state(cpu_fd, &mshv_state);
-    if (ret == 0) {
-        memcpy(state, buffer, sizeof(*state));
-    }
-    qemu_vfree(buffer);
-    if (ret < 0) {
-        error_report("failed to get lapic");
-        return -1;
-    }
-
-    return 0;
-}
-
 static uint32_t set_apic_delivery_mode(uint32_t reg, uint32_t mode)
 {
     return ((reg) & ~0x700) | ((mode) << 8);
 }
 
-static int set_vp_state(int cpu_fd, const struct mshv_get_set_vp_state *state)
-{
-    int ret;
-
-    ret = ioctl(cpu_fd, MSHV_SET_VP_STATE, state);
-    if (ret < 0) {
-        error_report("failed to set partition state: %s", strerror(errno));
-        return -1;
-    }
-
-    return 0;
-}
-
-static int set_lapic(const CPUState *cpu,
-                     const struct hv_local_interrupt_controller_state *state)
-{
-    int ret;
-    size_t size = 4096;
-    /* buffer aligned to 4k, as *state requires that */
-    void *buffer = qemu_memalign(size, size);
-    struct mshv_get_set_vp_state mshv_state = { 0 };
-    int cpu_fd = mshv_vcpufd(cpu);
-
-    if (!state) {
-        error_report("lapic state is NULL");
-        return -1;
-    }
-    memcpy(buffer, state, sizeof(*state));
-
-    mshv_state.buf_ptr = (uint64_t) buffer;
-    mshv_state.buf_sz = size;
-    mshv_state.type = MSHV_VP_STATE_LAPIC;
-
-    ret = set_vp_state(cpu_fd, &mshv_state);
-    qemu_vfree(buffer);
-    if (ret < 0) {
-        error_report("failed to set lapic: %s", strerror(errno));
-        return -1;
-    }
-
-    return 0;
-}
-
 static int init_lint(const CPUState *cpu)
 {
     int ret;
     uint32_t *lvt_lint0, *lvt_lint1;
+    int cpu_fd = mshv_vcpufd(cpu);
 
     struct hv_local_interrupt_controller_state lapic_state = { 0 };
-    ret = get_lapic(cpu, &lapic_state);
+    ret = mshv_get_lapic(cpu_fd, &lapic_state);
     if (ret < 0) {
         return ret;
     }
@@ -1026,7 +944,7 @@ static int init_lint(const CPUState *cpu)
 
     /* TODO: should we skip setting lapic if the values are the same? */
 
-    return set_lapic(cpu, &lapic_state);
+    return mshv_set_lapic(cpu_fd, &lapic_state);
 }
 
 int mshv_arch_store_vcpu_state(const CPUState *cpu)
Signature · 1 line
2.34.1
#16 lore raw

PATCH 16/34 target/i386/mshv: migrate LAPIC state

In reply to Magnus Kulke

This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.

The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.

An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare VMSTATE_BUFFER for the hv_lapic_state field.

In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.

---
 hw/intc/apic_common.c           |  3 ++
 include/hw/i386/apic_internal.h |  5 +++
 target/i386/mshv/mshv-cpu.c     | 61 +++++++++++++++++++++++++++++++--
 3 files changed, 67 insertions(+), 2 deletions(-)
hw/intc/apic_common.c +3 -0
index bf4abc21d7..a7df870f1a 100644
@@ -380,6 +380,9 @@ static const VMStateDescription vmstate_apic_common = {
         VMSTATE_INT64(next_time, APICCommonState),
         VMSTATE_INT64(timer_expiry,
                       APICCommonState), /* open-coded timer state */
+#ifdef CONFIG_MSHV
+        VMSTATE_BUFFER(hv_lapic_state, APICCommonState),
+#endif
         VMSTATE_END_OF_LIST()
     },
     .subsections = (const VMStateDescription * const []) {
include/hw/i386/apic_internal.h +5 -0
index 0cb06bbc76..6d4ccca4e8 100644
@@ -23,6 +23,7 @@
 
 #include "cpu.h"
 #include "hw/i386/apic.h"
+#include "hw/hyperv/hvgdk_mini.h"
 #include "system/memory.h"
 #include "qemu/timer.h"
 #include "target/i386/cpu-qom.h"
@@ -188,6 +189,10 @@ struct APICCommonState {
     DeviceState *vapic;
     hwaddr vapic_paddr; /* note: persistence via kvmvapic */
     uint32_t extended_log_dest;
+
+#ifdef CONFIG_MSHV
+    uint8_t hv_lapic_state[sizeof(struct hv_local_interrupt_controller_state)];
+#endif
 };
 
 typedef struct VAPICState {
target/i386/mshv/mshv-cpu.c +59 -2

93 lines not on this page · raw message

Signature · 1 line
2.34.1
#17 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 15 lines · Magnus Kulke

This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.

The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.

An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare VMSTATE_BUFFER for the hv_lapic_state field.

Quoted 5 lines

In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.

Hi,

Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?

Or is that not a wanted goal?

#18 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 19 lines · Mohamed Mediouni
Quoted 19 lines · Magnus Kulke

This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.

The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.

An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare VMSTATE_BUFFER for the hv_lapic_state field.

In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.

Quoted 9 lines
Quoted 1 line

Hi,

Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?

Or is that not a wanted goal?

yeah, for the time being it's a non-goal. there might a way to map hyperv lapic state onto existing state fields, but the mshv crates (which is the closest to a reference impl for mshv uapi) currently only expose the state as an opaque buffer.

#19 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

In reply to Mohamed Mediouni
Quoted 19 lines · Mohamed Mediouni
Quoted 19 lines · Magnus Kulke

This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.

The HyperV LAPIC is similar to the the split-irqchip in KVM, it will only handle MSI/X interrupts. PIC and IOAPIC have to be handled in userland.

An opaque blob is added to the APICCommonState, guarded behind a flag, hence it will be covered by a migration, as we declare VMSTATE_BUFFER for the hv_lapic_state field.

In the future we might want to introduce a dedicated class for MSHV, that would require us to wire up an IOAPIC delivery path to QEMU's userland emulation.

Quoted 9 lines
Quoted 1 line

Hi,

Is there a way to make this not opaque in a way that retains the ability to Save-restore between KVM and MSHV?

Or is that not a wanted goal?

as we discussed offline: there is mapping code in the mshv-ioctls crate and some prior art in WHPX that we can use as a reference. So we are to able to roundtrip the lapic state using generic field on APICCommonState and a few additional mshv-specific fields. This seems to work reliably in my migration tests.

I'll update this in the next iteration of this series. A KVM <=> MSHV migration path remains a non-goal, but it'll be a bit nicer to have the state transparent in the migration stream.

best,

magnus

maintainer
#20 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

In reply to Magnus Kulke

Il ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]> ha scritto:

Quoted 1 line

+#ifdef CONFIG_MSHV

+ VMSTATE_BUFFER(hv_lapic_state, APICCommonState),

+#endif

This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.

     DeviceState *vapic;
Quoted 8 lines

hwaddr vapic_paddr; /* note: persistence via kvmvapic */ uint32_t extended_log_dest;

include/hw/i386/apic_internal.h +3 -0
+
+#ifdef CONFIG_MSHV
+    uint8_t hv_lapic_state[sizeof(struct

hv_local_interrupt_controller_state)];
+#endif

No, this is called common for a reason. You need to implement a subclass and handle migration in there.

Paolo

#21 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 15 lines · Paolo Bonzini

Il ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]> ha scritto:

Quoted 1 line

+#ifdef CONFIG_MSHV

+ VMSTATE_BUFFER(hv_lapic_state, APICCommonState),

+#endif

This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.

     DeviceState *vapic;
Quoted 12 lines
Quoted 8 lines

hwaddr vapic_paddr; /* note: persistence via kvmvapic */ uint32_t extended_log_dest;

include/hw/i386/apic_internal.h +3 -0
+
+#ifdef CONFIG_MSHV
+    uint8_t hv_lapic_state[sizeof(struct

hv_local_interrupt_controller_state)];
+#endif

No, this is called common for a reason. You need to implement a subclass and handle migration in there.

makes sense. I think I found a way to put the additional fields into a subclass. the apic_id is semantic, so we cannot use the auto-wiring dc->vmds state, but APIC_COMMON is doing the same. I'll send a revised patch soon.

thanks,

magnus

maintainer
#22 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Il mer 1 lug 2026, 17:30 Magnus Kulke <[email protected]> ha scritto:

Quoted 30 lines · Paolo Bonzini
Quoted 1 line

Il ven 17 apr 2026, 12:57 Magnus Kulke <[email protected]>

ha

Quoted 14 lines

scritto:

Quoted 1 line

+#ifdef CONFIG_MSHV

+ VMSTATE_BUFFER(hv_lapic_state, APICCommonState),

+#endif

This breaks migration across releases for everyone else, and also across builds that have different --enable-mshv. Compile-time defines are almost always wrong in VMState definitions.

     DeviceState *vapic;
Quoted 12 lines
Quoted 8 lines

hwaddr vapic_paddr; /* note: persistence via kvmvapic */ uint32_t extended_log_dest;

include/hw/i386/apic_internal.h +3 -0
+
+#ifdef CONFIG_MSHV
+    uint8_t hv_lapic_state[sizeof(struct

hv_local_interrupt_controller_state)];
+#endif

No, this is called common for a reason. You need to implement a subclass and handle migration in there.

Quoted 2 lines

makes sense. I think I found a way to put the additional fields into a subclass.

What are the additional fields that aren't covered (i.e. cannot be copied from hv_lapic_state to the struct and back) by the common APIC state?

Paolo

the apic_id is semantic, so we cannot use the auto-wiring

Quoted 9 lines

dc->vmds state, but APIC_COMMON is doing the same. I'll send a revised patch
soon.

thanks,

magnus

#23 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 3 lines · Paolo Bonzini

What are the additional fields that aren't covered (i.e. cannot be copied from hv_lapic_state to the struct and back) by the common APIC state?

that would be those:

    uint32_t apic_version;
    uint32_t apic_lvt_lint0;
    uint32_t apic_lvt_lint1;
    uint32_t apic_lvt_cmci;
    uint32_t apic_error_status;
    uint32_t apic_counter_value;
    uint32_t apic_remote_read;

Reference struct is this: hv_local_interrupt_controller_state

https://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h

Notably we have to do this for lvt_lint0+1. We cannot use the respective fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,

pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()

For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion. Hence we roundtrip lint0/lint1 in a dedicated mshv struct.

best,

magnus

maintainer
#24 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Il gio 2 lug 2026, 11:49 Magnus Kulke <[email protected]> ha scritto:

Quoted 16 lines · Paolo Bonzini
Quoted 4 lines

What are the additional fields that aren't covered (i.e. cannot be copied from hv_lapic_state to the struct and back) by the common APIC state?

that would be those:

    uint32_t apic_version;
    uint32_t apic_lvt_lint0;
    uint32_t apic_lvt_lint1;
    uint32_t apic_lvt_cmci;
    uint32_t apic_error_status;
    uint32_t apic_counter_value;
    uint32_t apic_remote_read;
Quoted 3 lines

Reference struct is this: hv_local_interrupt_controller_state

https://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h

For anything that's new but part of the standard APIC you can place them in common APIC state, just making migration state part of a subsection.

Notably we have to do this for lvt_lint0+1. We cannot use the respective

Quoted 7 lines

fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,

pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()

For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion.

What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.

For LINT1, can you not inject an NMI either?

Thanks,

Paolo

Hence we roundtrip

Quoted 8 lines

lint0/lint1 in a dedicated mshv struct.

best,

magnus

#25 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 33 lines · Paolo Bonzini

Il gio 2 lug 2026, 11:49 Magnus Kulke <[email protected]> ha scritto:

Quoted 3 lines · Paolo Bonzini

What are the additional fields that aren't covered (i.e. cannot be copied from hv_lapic_state to the struct and back) by the common APIC state?

that would be those:

    uint32_t apic_version;
    uint32_t apic_lvt_lint0;
    uint32_t apic_lvt_lint1;
    uint32_t apic_lvt_cmci;
    uint32_t apic_error_status;
    uint32_t apic_counter_value;
    uint32_t apic_remote_read;

Reference struct is this: hv_local_interrupt_controller_state

https://gitlab.com/qemu/qemu/-/blob/master/include/hw/hyperv/hvgdk_mini.h

For anything that's new but part of the standard APIC you can place them in common APIC state, just making migration state part of a subsection.

Notably we have to do this for lvt_lint0+1. We cannot use the respective fields in the common APIC state, because writing to those has a side effect in QEMU: If unmasked, it will trigger a chain of events,

Quoted 6 lines

pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()

For MSHV (and WHPX afaiu) we're currently not able to trigger legacy interrupts from userspace in this fashion.

What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.

Hi,

For injecting LINT0, it’s possible on modern WHPX when using the hw irqchip
- starting from Server 2022/Windows 11 only - not before that point.

How it’s done is though cancelling the vCPU execution, wait until the vm exits as a result, set an interrupt window, run the vCPU again and then when it exits Manually injecting an event…

That’s unlike non-legacy interrupts where the request interrupts path actually works on Hyper-V.

Quoted 2 lines

For LINT1, can you not inject an NMI either?

The request interrupt path works at least to some extent on LINT1/NMIs unlike LINT0 but didn’t dig deeper to see if it has more issues…

I’m convinced that an Hyper-V specific structure is the wrong idea for this though...

Quoted 12 lines

Thanks,

Paolo

Hence we roundtrip
lint0/lint1 in a dedicated mshv struct.

best,

magnus

#26 lore raw

Re: [PATCH 16/34] target/i386/mshv: migrate LAPIC state

Quoted 16 lines · Mohamed Mediouni
Quoted 2 lines · Paolo Bonzini

What is the issue? The interrupt won't be triggered either if you use a separate field, or if you use the normal one.

Hi,

For injecting LINT0, it’s possible on modern WHPX when using the hw irqchip
- starting from Server 2022/Windows 11 only - not before that point.

How it’s done is though cancelling the vCPU execution, wait until the vm exits as a result, set an interrupt window, run the vCPU again and then when it exits Manually injecting an event…

That’s unlike non-legacy interrupts where the request interrupts path actually works on Hyper-V.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

For LINT1, can you not inject an NMI either?

The request interrupt path works at least to some extent on LINT1/NMIs unlike LINT0 but didn’t dig deeper to see if it has more issues…

I’m convinced that an Hyper-V specific structure is the wrong idea for this though...

Thanks for chiming in Mohamed. Indeed, I think the culprit is that injecting legacy interrupts is currently not working as expected on MSHV either, so this behaviour is probably related to Microsoft Hypervisor itself.

When you refer to "cancelling the vCPU exection", I suspect you mean manipulating the intercept/explicit suspend registers? I understand that the mshv driver maintainers would prefer not to expose that particular knob, since the kernel driver maintains some internal suspend state. Thus the hvcall is blocked from dom0 userspace currently. I'll have to further clarify that with the mshv team, though.

Mirroring the LINT0/1 fields in a dedicated MsvhApic structure was motivated by the legacy interrupt injection path not working and thus breaking the destination on a migration. Assuming we don't have the option to make it work without a kernel change (in the short term), what would be your suggestion?

I can image we can short-circuit this chain for mshv, is that preferable?

pic_irq_request() => apic_accept_pic_intr() => apic_deliver_pic_intr()

best,

magnus

#27 lore raw

PATCH 17/34 target/i386/mshv: move msr code to arch

In reply to Magnus Kulke

The MSR code is x86 specific, hence it's better suited in the arch tree.

---
 accel/mshv/meson.build            | 1 -
 target/i386/mshv/meson.build      | 1 +
 {accel => target/i386}/mshv/msr.c | 0
 3 files changed, 1 insertion(+), 1 deletion(-)
 rename {accel => target/i386}/mshv/msr.c (100%)
accel/mshv/meson.build +0 -1
index c1b1787c5e..e433187cde 100644
@@ -1,6 +1,5 @@
 system_ss.add(when: 'CONFIG_MSHV', if_true: files(
   'irq.c',
   'mem.c',
-  'msr.c',
   'mshv-all.c'
 ))
target/i386/mshv/meson.build +1 -0
index 5eb6e833a6..f44e84688d 100644
@@ -3,6 +3,7 @@ i386_mshv_ss = ss.source_set()
 i386_mshv_ss.add(files(
   'mshv-apic.c',
   'mshv-cpu.c',
+  'msr.c',
 ))
 
 i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
target/i386/mshv/msr.c renamed from accel/mshv/msr.c +0 -0
similarity index 100%
rename from accel/mshv/msr.c
rename to target/i386/mshv/msr.c
Signature · 1 line
2.34.1
#28 lore raw

PATCH 18/34 accel/mshv: store partition proc features

In reply to Magnus Kulke

We retrieve and store processor features on the state, so we can query them later when deciding which MSRs to migrate.

---
 accel/mshv/mshv-all.c     |  57 +++++++++++++++
 include/hw/hyperv/hvhdk.h | 150 ++++++++++++++++++++++++++++++++++++++
 include/system/mshv_int.h |   2 +
 3 files changed, 209 insertions(+)
accel/mshv/mshv-all.c +57 -0
index 056b19b3b8..4c1a42d002 100644
@@ -106,6 +106,57 @@ static int resume_vm(int vm_fd)
     return 0;
 }
 
+static int get_partition_property(int vm_fd, uint32_t feature_bank,
+                                  uint64_t *value)
+{
+    struct hv_input_get_partition_property in = {0};
+    struct hv_output_get_partition_property out = {0};
+    struct mshv_root_hvcall args = {0};
+    int ret;
+
+    in.property_code = feature_bank;
+
+    args.code    = HVCALL_GET_PARTITION_PROPERTY;
+    args.in_sz   = sizeof(in);
+    args.in_ptr  = (uint64_t)&in;
+    args.out_sz  = sizeof(out);
+    args.out_ptr = (uint64_t)&out;
+
+    ret = ioctl(vm_fd, MSHV_ROOT_HVCALL, &args);
+    if (ret < 0) {
+        error_report("Failed to get guest partition property bank: %s",
+                     strerror(errno));
+        return -1;
+    }
+
+    *value = out.property_value;
+    return 0;
+}
+
+static int get_proc_features(int vm_fd,
+                             union hv_partition_processor_features *features)
+{
+    int ret;
+
+    ret = get_partition_property(vm_fd,
+                                 HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0,
+                                 features[0].as_uint64);
+    if (ret < 0) {
+        error_report("Failed to get processor features bank 0");
+        return -1;
+    }
+
+    ret = get_partition_property(vm_fd,
+                                 HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1,
+                                 features[1].as_uint64);
+    if (ret < 0) {
+        error_report("Failed to get processor features bank 1");
+        return -1;
+    }
+
+    return 0;
+}
+
 static int create_partition(int mshv_fd, int *vm_fd)
 {
     int ret;
@@ -441,6 +492,12 @@ static int mshv_init(AccelState *as, MachineState *ms)
 
     s->vm = vm_fd;
     s->fd = mshv_fd;
+
+    ret = get_proc_features(vm_fd, &s->processor_features);
+    if (ret < 0) {
+        return -1;
+    }
+
     s->nr_as = 1;
     s->as = g_new0(MshvAddressSpace, s->nr_as);
 
include/hw/hyperv/hvhdk.h +150 -0

162 lines not on this page · raw message

include/system/mshv_int.h +2 -0
index f86c7a3be6..2b6d7b2f35 100644
@@ -15,6 +15,7 @@
 #define QEMU_MSHV_INT_H
 
 #define MSHV_MSR_ENTRIES_COUNT 64
+#include "hw/hyperv/hvhdk.h"
 
 struct mshv_get_set_vp_state;
 
@@ -55,6 +56,7 @@ struct MshvState {
     int nr_allocated_irq_routes;
     unsigned long *used_gsi_bitmap;
     unsigned int gsi_count;
+    union hv_partition_processor_features processor_features;
 };
 
 typedef struct MshvMsiControl {
Signature · 1 line
2.34.1
#29 lore raw

PATCH 19/34 target/i386/mshv: expose msvh_get_generic_regs

In reply to Magnus Kulke

We expose the fn, so we can call them from the other source files (msr.c).

---
 include/system/mshv_int.h   |  2 ++
 target/i386/mshv/mshv-cpu.c | 15 ++++++---------
 2 files changed, 8 insertions(+), 9 deletions(-)
include/system/mshv_int.h +2 -0
index 2b6d7b2f35..2c5d16bf9a 100644
@@ -81,6 +81,8 @@ int mshv_configure_vcpu(const CPUState *cpu);
 int mshv_run_vcpu(int vm_fd, CPUState *cpu, hv_message *msg, MshvVmExit *exit);
 int mshv_set_generic_regs(const CPUState *cpu, const hv_register_assoc *assocs,
                           size_t n_regs);
+int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,
+                          size_t n_regs);
 int mshv_arch_store_vcpu_state(const CPUState *cpu);
 int mshv_arch_load_vcpu_state(CPUState *cpu);
 void mshv_arch_init_vcpu(CPUState *cpu);
target/i386/mshv/mshv-cpu.c +6 -9
index 22c962c5ac..2d1f90637e 100644
@@ -108,9 +108,6 @@ static enum hv_register_name FPU_REGISTER_NAMES[26] = {
 };
 
 static int set_special_regs(const CPUState *cpu);
-static int get_generic_regs(CPUState *cpu,
-                            struct hv_register_assoc *assocs,
-                            size_t n_regs);
 
 static int get_lapic(CPUState *cpu)
 {
@@ -187,7 +184,7 @@ static int get_fpu(CPUState *cpu)
     for (size_t i = 0; i < n_regs; i++) {
         assocs[i].name = FPU_REGISTER_NAMES[i];
     }
-    ret = get_generic_regs(cpu, assocs, n_regs);
+    ret = mshv_get_generic_regs(cpu, assocs, n_regs);
     if (ret < 0) {
         error_report("failed to get special registers");
         return -errno;
@@ -207,7 +204,7 @@ static int get_xc_reg(CPUState *cpu)
 
     assocs[0].name = HV_X64_REGISTER_XFEM;
 
-    ret = get_generic_regs(cpu, assocs, 1);
+    ret = mshv_get_generic_regs(cpu, assocs, 1);
     if (ret < 0) {
         error_report("failed to get xcr0");
         return -1;
@@ -300,8 +297,8 @@ int mshv_set_generic_regs(const CPUState *cpu, const hv_register_assoc *assocs,
     return 0;
 }
 
-static int get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,
-                            size_t n_regs)
+int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,
+                          size_t n_regs)
 {
     int cpu_fd = mshv_vcpufd(cpu);
     int vp_index = cpu->cpu_index;
@@ -450,7 +447,7 @@ static int get_standard_regs(CPUState *cpu)
     for (size_t i = 0; i < n_regs; i++) {
         assocs[i].name = STANDARD_REGISTER_NAMES[i];
     }
-    ret = get_generic_regs(cpu, assocs, n_regs);
+    ret = mshv_get_generic_regs(cpu, assocs, n_regs);
     if (ret < 0) {
         error_report("failed to get standard registers");
         return -1;
@@ -527,7 +524,7 @@ static int get_special_regs(CPUState *cpu)
     for (size_t i = 0; i < n_regs; i++) {
         assocs[i].name = SPECIAL_REGISTER_NAMES[i];
     }
-    ret = get_generic_regs(cpu, assocs, n_regs);
+    ret = mshv_get_generic_regs(cpu, assocs, n_regs);
     if (ret < 0) {
         error_report("failed to get special registers");
         return -errno;
Signature · 1 line
2.34.1
#30 lore raw

PATCH 20/34 target/i386/mshv: migrate MSRs

In reply to Magnus Kulke

In this change the we rewrite the existing MSR logic to make MSRs migratable:

- we map them on existing QEMU fields in the CPU. A table and a macro MSHV_ENV_FIELD is used to associate a HV register name to the their msr index and their offset in the cpu state struct. The list is not exhaustive and will be extended in follow-up commits.
- mshv_set/get_msrs() fns are called in the arch_load/store_vcpu_state() fns. they use use generic registers ioctl's and map the input/output via load/store_to/from_env() from/to the hv register content to the cpu state representation.
- init_msrs() has been moved from mshv-vcpu to the msr source file
- we need to perform some filtering of MSR because before writing and reading, because the hvcalls will fail if the partition doesn't support a given MSRs.
- Some MSRs are partition-wide and so we will only write the to on the BSP.

---
 include/hw/hyperv/hvgdk_mini.h |  16 +
 include/system/mshv_int.h      |  17 +-
 target/i386/mshv/mshv-cpu.c    |  40 +--
 target/i386/mshv/msr.c         | 554 ++++++++++++++-------------------
 4 files changed, 259 insertions(+), 368 deletions(-)
include/hw/hyperv/hvgdk_mini.h +16 -0
index c3a8f33280..4d1e062e48 100644
@@ -9,6 +9,19 @@
 
 #define MSHV_IOCTL  0xB8
 
+/* Hyper-V specific model specific registers (MSRs) */
+
+/* HV_X64_SYNTHETIC_MSR */
+#define HV_X64_MSR_GUEST_OS_ID      0x40000000
+#define HV_X64_MSR_HYPERCALL        0x40000001
+#define HV_X64_MSR_VP_INDEX         0x40000002
+#define HV_X64_MSR_RESET            0x40000003
+#define HV_X64_MSR_VP_RUNTIME       0x40000010
+#define HV_X64_MSR_TIME_REF_COUNT   0x40000020
+#define HV_X64_MSR_REFERENCE_TSC    0x40000021
+#define HV_X64_MSR_TSC_FREQUENCY    0x40000022
+#define HV_X64_MSR_APIC_FREQUENCY   0x40000023
+
 typedef enum hv_register_name {
     /* Pending Interruption Register */
     HV_REGISTER_PENDING_INTERRUPTION = 0x00010002,
@@ -152,12 +165,14 @@ typedef enum hv_register_name {
     /* Available */
 
     HV_X64_REGISTER_SPEC_CTRL       = 0x00080084,
+    HV_X64_REGISTER_TSC_DEADLINE    = 0x00080095,
     HV_X64_REGISTER_TSC_ADJUST      = 0x00080096,
 
     /* Other MSRs */
     HV_X64_REGISTER_MSR_IA32_MISC_ENABLE = 0x000800A0,
 
     /* Misc */
+    HV_X64_REGISTER_HYPERCALL       = 0x00090001,
     HV_REGISTER_GUEST_OS_ID         = 0x00090002,
     HV_REGISTER_REFERENCE_TSC       = 0x00090017,
 
@@ -797,6 +812,7 @@ struct hv_cpuid {
 #define IA32_MSR_DEBUG_CTL        0x1D9
 #define IA32_MSR_SPEC_CTRL        0x00000048
 #define IA32_MSR_TSC_ADJUST       0x0000003b
+#define IA32_MSR_TSC_DEADLINE     0x000006e0
 
 #define IA32_MSR_MISC_ENABLE 0x000001a0
 
include/system/mshv_int.h +3 -14
index 2c5d16bf9a..29b363e73e 100644
@@ -14,7 +14,6 @@
 #ifndef QEMU_MSHV_INT_H
 #define QEMU_MSHV_INT_H
 
-#define MSHV_MSR_ENTRIES_COUNT 64
 #include "hw/hyperv/hvhdk.h"
 
 struct mshv_get_set_vp_state;
@@ -116,18 +115,8 @@ void mshv_set_phys_mem(MshvMemoryListener *mml, MemoryRegionSection *section,
                        bool add);
 
 /* msr */
-typedef struct MshvMsrEntry {
-  uint32_t index;
-  uint32_t reserved;
-  uint64_t data;
-} MshvMsrEntry;
-
-typedef struct MshvMsrEntries {
-    MshvMsrEntry entries[MSHV_MSR_ENTRIES_COUNT];
-    uint32_t nmsrs;
-} MshvMsrEntries;
-
-int mshv_configure_msr(const CPUState *cpu, const MshvMsrEntry *msrs,
-                       size_t n_msrs);
+int mshv_init_msrs(const CPUState *cpu);
+int mshv_get_msrs(CPUState *cpu);
+int mshv_set_msrs(const CPUState *cpu);
 
 #endif
target/i386/mshv/mshv-cpu.c +12 -28
index 2d1f90637e..760fcfe8da 100644
@@ -580,6 +580,11 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)
         return ret;
     }
 
+    ret = mshv_get_msrs(cpu);
+    if (ret < 0) {
+        return ret;
+    }
+
     return 0;
 }
 
@@ -1025,6 +1030,12 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)
         return ret;
     }
 
+    /* INVARIANT: LAPIC must be restored before MSRs (TSC_DEADLINE) */
+    ret = mshv_set_msrs(cpu);
+    if (ret < 0) {
+        return ret;
+    }
+
     return 0;
 }
 
@@ -1511,33 +1522,6 @@ void mshv_init_mmio_emu(void)
     init_emu(&mshv_x86_emul_ops);
 }
 
-static int init_msrs(const CPUState *cpu)
-{
-    int ret;
-    uint64_t d_t = MSR_MTRR_ENABLE | MSR_MTRR_MEM_TYPE_WB;
-
-    const struct hv_register_assoc assocs[] = {
-        { .name = HV_X64_REGISTER_SYSENTER_CS,       .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_SYSENTER_ESP,      .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_SYSENTER_EIP,      .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_STAR,              .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_CSTAR,             .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_LSTAR,             .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_KERNEL_GS_BASE,    .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_SFMASK,            .value.reg64 = 0x0 },
-        { .name = HV_X64_REGISTER_MSR_MTRR_DEF_TYPE, .value.reg64 = d_t },
-    };
-    QEMU_BUILD_BUG_ON(ARRAY_SIZE(assocs) > MSHV_MSR_ENTRIES_COUNT);
-
-    ret = mshv_set_generic_regs(cpu, assocs, ARRAY_SIZE(assocs));
-    if (ret < 0) {
-        error_report("failed to put msrs");
-        return -1;
-    }
-
-    return 0;
-}
-
 void mshv_arch_init_vcpu(CPUState *cpu)
 {
     X86CPU *x86_cpu = X86_CPU(cpu);
@@ -1567,7 +1551,7 @@ void mshv_arch_init_vcpu(CPUState *cpu)
     ret = init_cpuid2(cpu);
     assert(ret == 0);
 
-    ret = init_msrs(cpu);
+    ret = mshv_init_msrs(cpu);
     assert(ret == 0);
 
     ret = init_lint(cpu);
target/i386/mshv/msr.c +228 -326

590 lines not on this page · raw message

Signature · 1 line
2.34.1
#31 lore raw

PATCH 21/34 target/i386/mshv: migrate MTRR MSRs

In reply to Magnus Kulke

This change roundtrips memory access/caching MSRs. The mapping scheme is a bit more elaborate on these, so we have added a special handling instead of individual entries in the MSR mapping table.

---
 target/i386/mshv/msr.c | 136 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 129 insertions(+), 7 deletions(-)
target/i386/mshv/msr.c +129 -7
index 0ecd864458..b26375e4c2 100644
@@ -79,6 +79,10 @@ static const MshvMsrEnvMap msr_env_map[] = {
     { HV_X64_MSR_SIMP,      HV_REGISTER_SIMP,
                             offsetof(CPUX86State, msr_hv_synic_msg_page) },
 
+    /* MTRR default type */
+    { IA32_MSR_MTRR_DEF_TYPE, HV_X64_REGISTER_MSR_MTRR_DEF_TYPE,
+                              offsetof(CPUX86State, mtrr_deftype) },
+
     /* Other */
 
     /* TODO: find out processor features that correlate to unsupported MSRs. */
@@ -90,6 +94,98 @@ static const MshvMsrEnvMap msr_env_map[] = {
                             offsetof(CPUX86State, spec_ctrl) },
 };
 
+/*
+ * The assocs have to be set according to this schema:
+ *      8  entries for 0-7 mtrr_base
+ *      8  entries for mtrr_mask 0-7
+ *      11 entries for 1 x 64k, 2 x 16k, 8 x 4k fixed MTRR
+ *      27 total entries
+ */
+
+#define MSHV_MTRR_MSR_COUNT 27
+#define MSHV_MSR_TOTAL_COUNT (ARRAY_SIZE(msr_env_map) + MSHV_MTRR_MSR_COUNT)
+
+static void store_in_env_mtrr_phys(CPUState *cpu,
+                                   const struct hv_register_assoc *assocs,
+                                   size_t n_assocs)
+{
+    X86CPU *x86_cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86_cpu->env;
+    size_t i, fixed_offset;
+    hv_register_name hv_name;
+    uint64_t base, mask;
+
+    assert(n_assocs == MSHV_MTRR_MSR_COUNT);
+
+    for (i = 0; i < MSR_MTRRcap_VCNT; i++) {
+        hv_name = HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i;
+        assert(assocs[i].name == hv_name);
+        hv_name = HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i;
+        assert(assocs[i + MSR_MTRRcap_VCNT].name == hv_name);
+
+        base = assocs[i].value.reg64;
+        mask = assocs[i + MSR_MTRRcap_VCNT].value.reg64;
+        env->mtrr_var[i].base = base;
+        env->mtrr_var[i].mask = mask;
+    }
+
+    /* fixed 1x 64, 2x 16, 8x 4 kB */
+    fixed_offset = MSR_MTRRcap_VCNT * 2;
+    for (i = 0; i < 11; i++) {
+        hv_name = HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i;
+        assert(assocs[fixed_offset + i].name == hv_name);
+        env->mtrr_fixed[i] = assocs[fixed_offset + i].value.reg64;
+    }
+}
+
+/*
+ * The assocs have to be set according to this schema:
+ *      8  entries for 0-7 mtrr_base
+ *      8  entries for mtrr_mask 0-7
+ *      11 entries for 1 x 64k, 2 x 16k, 8 x 4k fixed MTRR
+ *      27 total entries
+ */
+static void load_from_env_mtrr_phys(const CPUState *cpu,
+                                    struct hv_register_assoc *assocs,
+                                    size_t n_assocs)
+{
+    X86CPU *x86_cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86_cpu->env;
+    size_t i, fixed_offset;
+    uint64_t base, mask, fixed_value;
+    hv_register_name base_name, mask_name, fixed_name;
+    hv_register_assoc *assoc;
+
+    assert(n_assocs == MSHV_MTRR_MSR_COUNT);
+
+    for (i = 0; i < MSR_MTRRcap_VCNT; i++) {
+        base = env->mtrr_var[i].base;
+        mask = env->mtrr_var[i].mask;
+
+        base_name = HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i;
+        mask_name = HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i;
+
+        assoc = &assocs[i];
+        assoc->name = base_name;
+        assoc->value.reg64 = base;
+
+        assoc = &assocs[i + MSR_MTRRcap_VCNT];
+        assoc->name = mask_name;
+        assoc->value.reg64 = mask;
+    }
+
+    /* fixed 1x 64, 2x 16, 8x 4 kB */
+    fixed_offset = MSR_MTRRcap_VCNT * 2;
+    for (i = 0; i < 11; i++) {
+        fixed_name = HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i;
+        fixed_value = env->mtrr_fixed[i];
+
+        assoc = &assocs[fixed_offset + i];
+        assoc->name = fixed_name;
+        assoc->value.reg64 = fixed_value;
+    }
+}
+
 int mshv_init_msrs(const CPUState *cpu)
 {
     int ret;
@@ -131,8 +227,9 @@ static void store_in_env(CPUState *cpu, const struct hv_register_assoc *assocs,
     union hv_register_value hv_value;
     ptrdiff_t offset;
     uint32_t hv_name;
+    size_t mtrr_index;
 
-    assert(n_assocs <= (ARRAY_SIZE(msr_env_map)));
+    assert(n_assocs <= MSHV_MSR_TOTAL_COUNT);
 
     for (i = 0, j = 0; i < ARRAY_SIZE(msr_env_map); i++) {
         hv_name = assocs[j].name;
@@ -146,17 +243,38 @@ static void store_in_env(CPUState *cpu, const struct hv_register_assoc *assocs,
         MSHV_ENV_FIELD(env, offset) = hv_value.reg64;
         j++;
     }
+
+    mtrr_index = j;
+    store_in_env_mtrr_phys(cpu, &assocs[mtrr_index], MSHV_MTRR_MSR_COUNT);
 }
 
 static void set_hv_name_in_assocs(struct hv_register_assoc *assocs,
                                   size_t n_assocs)
 {
     size_t i;
+    size_t mtrr_offset, mtrr_fixed_offset;
+    hv_register_name hv_name;
+
+    assert(n_assocs == MSHV_MSR_TOTAL_COUNT);
 
-    assert(n_assocs == ARRAY_SIZE(msr_env_map));
     for (i = 0; i < ARRAY_SIZE(msr_env_map); i++) {
         assocs[i].name = msr_env_map[i].hv_name;
     }
+
+    mtrr_offset = ARRAY_SIZE(msr_env_map);
+    for (i = 0; i < MSR_MTRRcap_VCNT; i++) {
+        hv_name = HV_X64_REGISTER_MSR_MTRR_PHYS_BASE0 + i;
+        assocs[mtrr_offset + i].name = hv_name;
+        hv_name = HV_X64_REGISTER_MSR_MTRR_PHYS_MASK0 + i;
+        assocs[mtrr_offset + MSR_MTRRcap_VCNT + i].name = hv_name;
+    }
+
+    /* fixed 1x 64, 2x 16, 8x 4 kB */
+    mtrr_fixed_offset = mtrr_offset + MSR_MTRRcap_VCNT * 2;
+    for (i = 0; i < 11; i++) {
+        hv_name = HV_X64_REGISTER_MSR_MTRR_FIX64K00000 + i;
+        assocs[mtrr_fixed_offset + i].name = hv_name;
+    }
 }
 
 static bool msr_supported(uint32_t name)
@@ -181,8 +299,8 @@ static bool msr_supported(uint32_t name)
 int mshv_get_msrs(CPUState *cpu)
 {
     int ret = 0;
-    size_t n_assocs = ARRAY_SIZE(msr_env_map);
-    struct hv_register_assoc assocs[ARRAY_SIZE(msr_env_map)];
+    size_t n_assocs = MSHV_MSR_TOTAL_COUNT;
+    struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT];
     size_t i, j;
     uint32_t name;
 
@@ -223,8 +341,9 @@ static void load_from_env(const CPUState *cpu, struct hv_register_assoc *assocs,
     CPUX86State *env = &x86_cpu->env;
     ptrdiff_t offset;
     union hv_register_value *hv_value;
+    size_t mtrr_offset;
 
-    assert(n_assocs == ARRAY_SIZE(msr_env_map));
+    assert(n_assocs == MSHV_MSR_TOTAL_COUNT);
 
     for (i = 0; i < ARRAY_SIZE(msr_env_map); i++) {
         mapping = &msr_env_map[i];
@@ -233,12 +352,15 @@ static void load_from_env(const CPUState *cpu, struct hv_register_assoc *assocs,
         hv_value = &assocs[i].value;
         hv_value->reg64 = MSHV_ENV_FIELD(env, offset);
     }
+
+    mtrr_offset = ARRAY_SIZE(msr_env_map);
+    load_from_env_mtrr_phys(cpu, &assocs[mtrr_offset], MSHV_MTRR_MSR_COUNT);
 }
 
 int mshv_set_msrs(const CPUState *cpu)
 {
-    size_t n_assocs = ARRAY_SIZE(msr_env_map);
-    struct hv_register_assoc assocs[ARRAY_SIZE(msr_env_map)];
+    size_t n_assocs = MSHV_MSR_TOTAL_COUNT;
+    struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT];
     int ret;
     size_t i, j;
 
Signature · 1 line
2.34.1
#32 lore raw

PATCH 22/34 target/i386/mshv: migrate Synic SINT MSRs

In reply to Magnus Kulke

Migrate HyperV SynIC SINT MSRs. We can only read/write those if SCONTROL is enabled in the guest, hence we have to split the SINT MSR out and make reading/writing them dependent on that MSR.

---
 target/i386/mshv/msr.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
target/i386/mshv/msr.c +41 -0
index b26375e4c2..4881e15f4b 100644
@@ -303,6 +303,8 @@ int mshv_get_msrs(CPUState *cpu)
     struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT];
     size_t i, j;
     uint32_t name;
+    X86CPU *x86cpu = X86_CPU(cpu);
+    bool synic_enabled;
 
     set_hv_name_in_assocs(assocs, n_assocs);
 
@@ -329,6 +331,27 @@ int mshv_get_msrs(CPUState *cpu)
 
     store_in_env(cpu, assocs, n_assocs);
 
+    /* Read SINT MSRs only if SynIC is enabled */
+    synic_enabled = x86cpu->env.msr_hv_synic_control & 1;
+    if (synic_enabled) {
+        QEMU_BUILD_BUG_ON(MSHV_MSR_TOTAL_COUNT < HV_SINT_COUNT);
+
+        for (i = 0; i < HV_SINT_COUNT; i++) {
+            assocs[i].name = HV_REGISTER_SINT0 + i;
+        }
+
+        ret = mshv_get_generic_regs(cpu, assocs, HV_SINT_COUNT);
+        if (ret < 0) {
+            error_report("Failed to get SynIC SINT MSRs");
+            return -errno;
+        }
+
+        for (i = 0; i < HV_SINT_COUNT; i++) {
+            uint64_t hv_sint_value = assocs[i].value.reg64;
+            x86cpu->env.msr_hv_synic_sint[i] = hv_sint_value;
+        }
+    }
+
     return 0;
 }
 
@@ -363,6 +386,8 @@ int mshv_set_msrs(const CPUState *cpu)
     struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT];
     int ret;
     size_t i, j;
+    X86CPU *x86cpu = X86_CPU(cpu);
+    bool synic_enabled = x86cpu->env.msr_hv_synic_control & 1;
 
     load_from_env(cpu, assocs, n_assocs);
 
@@ -395,5 +420,21 @@ int mshv_set_msrs(const CPUState *cpu)
         return -errno;
     }
 
+    /* SINT MSRs can only be written if SCONTROL has been set, so we split */
+    if (synic_enabled) {
+        QEMU_BUILD_BUG_ON(MSHV_MSR_TOTAL_COUNT < HV_SINT_COUNT);
+
+        for (i = 0; i < HV_SINT_COUNT; i++) {
+            assocs[i].name        = HV_REGISTER_SINT0 + i;
+            assocs[i].value.reg64 = x86cpu->env.msr_hv_synic_sint[i];
+        }
+
+        ret = mshv_set_generic_regs(cpu, assocs, HV_SINT_COUNT);
+        if (ret < 0) {
+            error_report("Failed to set SynIC SINT MSRs");
+            return -errno;
+        }
+    }
+
     return 0;
 }
Signature · 1 line
2.34.1
#33 lore raw

PATCH 23/34 target/i386/mshv: migrate CET/SS MSRs

In reply to Magnus Kulke

This change migrates the MSRs required for CET shadow stack and indirect branch tracking. They are gated behind cet_ss_support || cet_ibt_support mshv processor feature flags.

---
 include/hw/hyperv/hvgdk_mini.h | 11 +++++++++++
 target/i386/mshv/msr.c         | 30 ++++++++++++++++++++++++++++++
 2 files changed, 41 insertions(+)
include/hw/hyperv/hvgdk_mini.h +11 -0
index 4d1e062e48..75e953b86c 100644
@@ -168,6 +168,17 @@ typedef enum hv_register_name {
     HV_X64_REGISTER_TSC_DEADLINE    = 0x00080095,
     HV_X64_REGISTER_TSC_ADJUST      = 0x00080096,
 
+    /* CET / Shadow Stack */
+    HV_X64_REGISTER_U_XSS                    = 0x0008008B,
+    HV_X64_REGISTER_U_CET                    = 0x0008008C,
+    HV_X64_REGISTER_S_CET                    = 0x0008008D,
+    HV_X64_REGISTER_SSP                      = 0x0008008E,
+    HV_X64_REGISTER_PL0_SSP                  = 0x0008008F,
+    HV_X64_REGISTER_PL1_SSP                  = 0x00080090,
+    HV_X64_REGISTER_PL2_SSP                  = 0x00080091,
+    HV_X64_REGISTER_PL3_SSP                  = 0x00080092,
+    HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR = 0x00080093,
+
     /* Other MSRs */
     HV_X64_REGISTER_MSR_IA32_MISC_ENABLE = 0x000800A0,
 
target/i386/mshv/msr.c +30 -0
index 4881e15f4b..b985500797 100644
@@ -83,6 +83,26 @@ static const MshvMsrEnvMap msr_env_map[] = {
     { IA32_MSR_MTRR_DEF_TYPE, HV_X64_REGISTER_MSR_MTRR_DEF_TYPE,
                               offsetof(CPUX86State, mtrr_deftype) },
 
+    /* CET / Shadow Stack */
+    { MSR_IA32_U_CET,       HV_X64_REGISTER_U_CET,
+                            offsetof(CPUX86State, u_cet) },
+    { MSR_IA32_S_CET,       HV_X64_REGISTER_S_CET,
+                            offsetof(CPUX86State, s_cet) },
+    { MSR_IA32_PL0_SSP,     HV_X64_REGISTER_PL0_SSP,
+                            offsetof(CPUX86State, pl0_ssp) },
+    { MSR_IA32_PL1_SSP,     HV_X64_REGISTER_PL1_SSP,
+                            offsetof(CPUX86State, pl1_ssp) },
+    { MSR_IA32_PL2_SSP,     HV_X64_REGISTER_PL2_SSP,
+                            offsetof(CPUX86State, pl2_ssp) },
+    { MSR_IA32_PL3_SSP,     HV_X64_REGISTER_PL3_SSP,
+                            offsetof(CPUX86State, pl3_ssp) },
+    { MSR_IA32_INT_SSP_TAB, HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR,
+                            offsetof(CPUX86State, int_ssp_table) },
+
+    /* XSAVE Supervisor State */
+    { MSR_IA32_XSS,         HV_X64_REGISTER_U_XSS,
+                            offsetof(CPUX86State, xss) },
+
     /* Other */
 
     /* TODO: find out processor features that correlate to unsupported MSRs. */
@@ -291,6 +311,16 @@ static bool msr_supported(uint32_t name)
         return mshv_state->processor_features.tsc_adjust_support;
     case HV_X64_REGISTER_TSC_DEADLINE:
         return mshv_state->processor_features.tsc_deadline_tmr_support;
+    case HV_X64_REGISTER_U_CET:
+    case HV_X64_REGISTER_S_CET:
+    case HV_X64_REGISTER_PL0_SSP:
+    case HV_X64_REGISTER_PL1_SSP:
+    case HV_X64_REGISTER_PL2_SSP:
+    case HV_X64_REGISTER_PL3_SSP:
+    case HV_X64_REGISTER_INTERRUPT_SSP_TABLE_ADDR:
+    case HV_X64_REGISTER_U_XSS:
+        return mshv_state->processor_features.cet_ss_support ||
+               mshv_state->processor_features.cet_ibt_support;
     }
 
     return true;
Signature · 1 line
2.34.1
#34 lore raw

PATCH 24/34 target/i386/mshv: migrate SIMP and SIEFP state

In reply to Magnus Kulke

This part SynIC state is retrieved from the hypervisor via aligned state pages:

- Add new synic source file
- Centralize the synic_enabled() check
- r/w pages from the hyper via aligned pages
- only handle pages when synic is enabled
- add buffers for migration to VM state

---
 include/system/mshv_int.h    |   7 ++
 target/i386/cpu.h            |   5 ++
 target/i386/machine.c        |  26 ++++++
 target/i386/mshv/meson.build |   1 +
 target/i386/mshv/mshv-cpu.c  |  64 +++++++++++++++
 target/i386/mshv/msr.c       |   7 +-
 target/i386/mshv/synic.c     | 155 +++++++++++++++++++++++++++++++++++
 7 files changed, 260 insertions(+), 5 deletions(-)
 create mode 100644 target/i386/mshv/synic.c
include/system/mshv_int.h +7 -0
index 29b363e73e..80df4030c5 100644
@@ -119,4 +119,11 @@ int mshv_init_msrs(const CPUState *cpu);
 int mshv_get_msrs(CPUState *cpu);
 int mshv_set_msrs(const CPUState *cpu);
 
+/* synic */
+int mshv_get_simp(int cpu_fd, uint8_t *page);
+int mshv_set_simp(int cpu_fd, const uint8_t *page);
+int mshv_get_siefp(int cpu_fd, uint8_t *page);
+int mshv_set_siefp(int cpu_fd, const uint8_t *page);
+bool mshv_synic_enabled(const CPUState *cpu);
+
 #endif
target/i386/cpu.h +5 -0
index 0b539155c4..d010d26146 100644
@@ -33,6 +33,7 @@
 #include "qemu/cpu-float.h"
 #include "qemu/timer.h"
 #include "standard-headers/asm-x86/kvm_para.h"
+#include "hw/hyperv/hvgdk_mini.h"
 
 #define XEN_NR_VIRQS 24
 
@@ -2291,6 +2292,10 @@ typedef struct CPUArchState {
 #if defined(CONFIG_HVF) || defined(CONFIG_MSHV) || defined(CONFIG_WHPX)
     void *emu_mmio_buf;
 #endif
+#if defined(CONFIG_MSHV)
+    uint8_t hv_simp_page[HV_HYP_PAGE_SIZE];
+    uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE];
+#endif
 
     uint64_t mcg_cap;
     uint64_t mcg_ctl;
target/i386/machine.c +26 -0
index 48a2a4b319..f94cc544b3 100644
@@ -952,6 +952,29 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = {
     }
 };
 
+#ifdef CONFIG_MSHV
+static bool mshv_synic_vp_state_needed(void *opaque)
+{
+    X86CPU *cpu = opaque;
+    CPUX86State *env = &cpu->env;
+
+    /* Only migrate SIMP/SIEFP if SynIC is enabled */
+    return env->msr_hv_synic_control & 1;
+}
+
+static const VMStateDescription vmstate_mshv_synic_vp_state = {
+    .name = "cpu/mshv_synic_vp_state",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = mshv_synic_vp_state_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_BUFFER(env.hv_simp_page, X86CPU),
+        VMSTATE_BUFFER(env.hv_siefp_page, X86CPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+#endif
+
 static bool avx512_needed(void *opaque)
 {
     X86CPU *cpu = opaque;
@@ -1916,6 +1939,9 @@ const VMStateDescription vmstate_x86_cpu = {
         &vmstate_cet,
 #ifdef TARGET_X86_64
         &vmstate_apx,
+#endif
+#ifdef CONFIG_MSHV
+        &vmstate_mshv_synic_vp_state,
 #endif
         NULL
     }
target/i386/mshv/meson.build +1 -0
index f44e84688d..a847a6c74c 100644
@@ -4,6 +4,7 @@ i386_mshv_ss.add(files(
   'mshv-apic.c',
   'mshv-cpu.c',
   'msr.c',
+  'synic.c',
 ))
 
 i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
target/i386/mshv/mshv-cpu.c +64 -0

88 lines not on this page · raw message

target/i386/mshv/msr.c +2 -5
index b985500797..a2d48249e9 100644
@@ -334,7 +334,6 @@ int mshv_get_msrs(CPUState *cpu)
     size_t i, j;
     uint32_t name;
     X86CPU *x86cpu = X86_CPU(cpu);
-    bool synic_enabled;
 
     set_hv_name_in_assocs(assocs, n_assocs);
 
@@ -362,8 +361,7 @@ int mshv_get_msrs(CPUState *cpu)
     store_in_env(cpu, assocs, n_assocs);
 
     /* Read SINT MSRs only if SynIC is enabled */
-    synic_enabled = x86cpu->env.msr_hv_synic_control & 1;
-    if (synic_enabled) {
+    if (mshv_synic_enabled(cpu)) {
         QEMU_BUILD_BUG_ON(MSHV_MSR_TOTAL_COUNT < HV_SINT_COUNT);
 
         for (i = 0; i < HV_SINT_COUNT; i++) {
@@ -417,7 +415,6 @@ int mshv_set_msrs(const CPUState *cpu)
     int ret;
     size_t i, j;
     X86CPU *x86cpu = X86_CPU(cpu);
-    bool synic_enabled = x86cpu->env.msr_hv_synic_control & 1;
 
     load_from_env(cpu, assocs, n_assocs);
 
@@ -451,7 +448,7 @@ int mshv_set_msrs(const CPUState *cpu)
     }
 
     /* SINT MSRs can only be written if SCONTROL has been set, so we split */
-    if (synic_enabled) {
+    if (mshv_synic_enabled(cpu)) {
         QEMU_BUILD_BUG_ON(MSHV_MSR_TOTAL_COUNT < HV_SINT_COUNT);
 
         for (i = 0; i < HV_SINT_COUNT; i++) {
target/i386/mshv/synic.c new file +155 -0

155 lines not on this page · raw message

Signature · 1 line
2.34.1
#35 lore raw

PATCH 25/34 target/i386/mshv: migrate STIMER state

In reply to Magnus Kulke

This part of Synic state is retrieved via a mem-aligned page. We declare the required space (size reference: rust-vmm/mshv) as a buffer on the VM state struct for inclusion in a migration.

Other than other SynIC features, STIMER doesn't depend on SCONTROL being set.

---
 include/system/mshv_int.h   |  2 ++
 target/i386/cpu.h           |  5 ++++
 target/i386/machine.c       | 20 +++++++++++++++
 target/i386/mshv/mshv-cpu.c | 12 +++++++++
 target/i386/mshv/synic.c    | 51 +++++++++++++++++++++++++++++++++++++
 5 files changed, 90 insertions(+)
include/system/mshv_int.h +2 -0
index 80df4030c5..7d685fc647 100644
@@ -125,5 +125,7 @@ int mshv_set_simp(int cpu_fd, const uint8_t *page);
 int mshv_get_siefp(int cpu_fd, uint8_t *page);
 int mshv_set_siefp(int cpu_fd, const uint8_t *page);
 bool mshv_synic_enabled(const CPUState *cpu);
+int mshv_get_synthetic_timers(int cpu_fd, uint8_t *state);
+int mshv_set_synthetic_timers(int cpu_fd, const uint8_t *state);
 
 #endif
target/i386/cpu.h +5 -0
index d010d26146..4ad4a35ce9 100644
@@ -45,6 +45,10 @@
 #define ELF_MACHINE_UNAME "i686"
 #endif
 
+#ifdef CONFIG_MSHV
+#define MSHV_STIMERS_STATE_SIZE 200
+#endif
+
 enum {
     R_EAX = 0,
     R_ECX = 1,
@@ -2295,6 +2299,7 @@ typedef struct CPUArchState {
 #if defined(CONFIG_MSHV)
     uint8_t hv_simp_page[HV_HYP_PAGE_SIZE];
     uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE];
+    uint8_t hv_synthetic_timers_state[MSHV_STIMERS_STATE_SIZE];
 #endif
 
     uint64_t mcg_cap;
target/i386/machine.c +20 -0
index f94cc544b3..38ccbbe19d 100644
@@ -10,6 +10,7 @@
 #include "exec/watchpoint.h"
 #include "system/kvm.h"
 #include "system/kvm_xen.h"
+#include "system/mshv.h"
 #include "system/tcg.h"
 
 #include "qemu/error-report.h"
@@ -953,6 +954,24 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = {
 };
 
 #ifdef CONFIG_MSHV
+
+static bool mshv_synthetic_timers_needed(void *opaque)
+{
+    /* Always migrate synthetic timers */
+    return mshv_enabled();
+}
+
+static const VMStateDescription vmstate_mshv_synthetic_timers = {
+    .name = "cpu/mshv_synthetic_timers",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = mshv_synthetic_timers_needed,
+    .fields = (const VMStateField[]) {
+        VMSTATE_BUFFER(env.hv_synthetic_timers_state, X86CPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static bool mshv_synic_vp_state_needed(void *opaque)
 {
     X86CPU *cpu = opaque;
@@ -1942,6 +1961,7 @@ const VMStateDescription vmstate_x86_cpu = {
 #endif
 #ifdef CONFIG_MSHV
         &vmstate_mshv_synic_vp_state,
+        &vmstate_mshv_synthetic_timers,
 #endif
         NULL
     }
target/i386/mshv/mshv-cpu.c +12 -0
index 36549857ae..9ce66e9aa1 100644
@@ -135,6 +135,12 @@ static int get_synic_state(CPUState *cpu)
     int cpu_fd = mshv_vcpufd(cpu);
     int ret;
 
+    ret = mshv_get_synthetic_timers(cpu_fd, env->hv_synthetic_timers_state);
+    if (ret < 0) {
+        error_report("failed to get synthetic timers");
+        return -1;
+    }
+
     /* SIMP/SIEFP can only be read when SynIC is enabled */
     if (!mshv_synic_enabled(cpu)) {
         return 0;
@@ -1039,6 +1045,12 @@ static int set_synic_state(const CPUState *cpu)
     int cpu_fd = mshv_vcpufd(cpu);
     int ret;
 
+    ret = mshv_set_synthetic_timers(cpu_fd, env->hv_synthetic_timers_state);
+    if (ret < 0) {
+        error_report("failed to set synthetic timers state");
+        return -1;
+    }
+
     /* SIMP/SIEFP can only be written when SynIC is enabled */
     if (!mshv_synic_enabled(cpu)) {
         return 0;
target/i386/mshv/synic.c +51 -0

57 lines not on this page · raw message

Signature · 1 line
2.34.1
#36 lore raw

PATCH 26/34 accel/mshv: introduce SaveVMHandler

In reply to Magnus Kulke

This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.

Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.

---
 accel/mshv/mshv-all.c | 75 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
accel/mshv/mshv-all.c +75 -0
index 4c1a42d002..5f81881c05 100644
@@ -39,6 +39,8 @@
 #include "system/mshv.h"
 #include "system/mshv_int.h"
 #include "system/reset.h"
+#include "migration/qemu-file-types.h"
+#include "migration/register.h"
 #include "trace.h"
 #include <err.h>
 #include <sys/ioctl.h>
@@ -47,6 +49,9 @@ bool mshv_allowed;
 
 MshvState *mshv_state;
 
+static int pause_vm(int vm_fd);
+static int resume_vm(int vm_fd);
+
 static int init_mshv(int *mshv_fd)
 {
     int fd = open("/dev/mshv", O_RDWR | O_CLOEXEC);
@@ -80,6 +85,64 @@ static int set_time_freeze(int vm_fd, int freeze)
     return 0;
 }
 
+static void mshv_save_state(QEMUFile *f, void *opaque)
+{
+    return;
+}
+
+static int mshv_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    return 0;
+}
+
+static int mshv_save_prepare(void *opaque, Error **errp)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = pause_vm(s->vm);
+    if (ret < 0) {
+        error_setg(errp, "Failed to pause VM for migration");
+        return -1;
+    }
+
+    return 0;
+}
+
+static void mshv_save_cleanup(void *opaque)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = resume_vm(s->vm);
+    if (ret < 0) {
+        error_report("Failed to resme VM after  migration");
+    }
+}
+
+static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = pause_vm(s->vm);
+    if (ret < 0) {
+        error_setg(errp, "Failed to pause VM for migration restore");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int mshv_load_cleanup(void *opaque)
+{
+    MshvState *s = opaque;
+
+    resume_vm(s->vm);
+
+    return 0;
+}
+
 static int pause_vm(int vm_fd)
 {
     int ret;
@@ -454,6 +517,15 @@ static int mshv_init_vcpu(CPUState *cpu)
     return 0;
 }
 
+static SaveVMHandlers savevm_mshv = {
+    .save_prepare = mshv_save_prepare,
+    .save_state = mshv_save_state,
+    .save_cleanup = mshv_save_cleanup,
+    .load_setup = mshv_load_setup,
+    .load_state = mshv_load_state,
+    .load_cleanup = mshv_load_cleanup,
+};
+
 static int mshv_init(AccelState *as, MachineState *ms)
 {
     MshvState *s;
@@ -509,6 +581,9 @@ static int mshv_init(AccelState *as, MachineState *ms)
                                   0, "mshv-memory");
     memory_listener_register(&mshv_io_listener, &address_space_io);
 
+    /* register custom handlers for migration events */
+    register_savevm_live("mshv", 0, 1, &savevm_mshv, s);
+
     return 0;
 }
 
Signature · 1 line
2.34.1
maintainer
#37 lore raw

Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler

Il ven 17 apr 2026, 12:58 Magnus Kulke <[email protected]> ha scritto:

Quoted 7 lines

This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.

Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.

Sorry, I forgot to reply about this one.

It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.

Paolo

Quoted 122 lines
---
 accel/mshv/mshv-all.c | 75 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 75 insertions(+)
accel/mshv/mshv-all.c +75 -1
index 4c1a42d002..5f81881c05 100644
@@ -39,6 +39,8 @@
 #include "system/mshv.h"
 #include "system/mshv_int.h"
 #include "system/reset.h"
+#include "migration/qemu-file-types.h"
+#include "migration/register.h"
 #include "trace.h"
 #include <err.h>
 #include <sys/ioctl.h>
@@ -47,6 +49,9 @@ bool mshv_allowed;
 
 MshvState *mshv_state;
 
+static int pause_vm(int vm_fd);
+static int resume_vm(int vm_fd);
+
 static int init_mshv(int *mshv_fd)
 {
     int fd = open("/dev/mshv", O_RDWR | O_CLOEXEC);
@@ -80,6 +85,64 @@ static int set_time_freeze(int vm_fd, int freeze)
     return 0;
 }
 
+static void mshv_save_state(QEMUFile *f, void *opaque)
+{
+    return;
+}
+
+static int mshv_load_state(QEMUFile *f, void *opaque, int version_id)
+{
+    return 0;
+}
+
+static int mshv_save_prepare(void *opaque, Error **errp)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = pause_vm(s->vm);
+    if (ret < 0) {
+        error_setg(errp, "Failed to pause VM for migration");
+        return -1;
+    }
+
+    return 0;
+}
+
+static void mshv_save_cleanup(void *opaque)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = resume_vm(s->vm);
+    if (ret < 0) {
+        error_report("Failed to resme VM after  migration");
+    }
+}
+
+static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)
+{
+    MshvState *s = opaque;
+    int ret;
+
+    ret = pause_vm(s->vm);
+    if (ret < 0) {
+        error_setg(errp, "Failed to pause VM for migration restore");
+        return -1;
+    }
+
+    return 0;
+}
+
+static int mshv_load_cleanup(void *opaque)
+{
+    MshvState *s = opaque;
+
+    resume_vm(s->vm);
+
+    return 0;
+}
+
 static int pause_vm(int vm_fd)
 {
     int ret;
@@ -454,6 +517,15 @@ static int mshv_init_vcpu(CPUState *cpu)
     return 0;
 }
 
+static SaveVMHandlers savevm_mshv = {
+    .save_prepare = mshv_save_prepare,
+    .save_state = mshv_save_state,
+    .save_cleanup = mshv_save_cleanup,
+    .load_setup = mshv_load_setup,
+    .load_state = mshv_load_state,
+    .load_cleanup = mshv_load_cleanup,
+};
+
 static int mshv_init(AccelState *as, MachineState *ms)
 {
     MshvState *s;
@@ -509,6 +581,9 @@ static int mshv_init(AccelState *as, MachineState *ms)
                                   0, "mshv-memory");
     memory_listener_register(&mshv_io_listener, &address_space_io);
 
+    /* register custom handlers for migration events */
+    register_savevm_live("mshv", 0, 1, &savevm_mshv, s);
+
     return 0;
 }
 
--

2.34.1

#38 lore raw

Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler

Quoted 11 lines · Paolo Bonzini

Il ven 17 apr 2026, 12:58 Magnus Kulke <[email protected]> ha scritto:

Quoted 7 lines

This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine.

Currently it's a skeleton that will just pause/resume the partition as part of a migration. It will later be extended with more specific steps like reference time and hypercall pages.

Quoted 8 lines

Sorry, I forgot to reply about this one.

It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.

Oh, good catch, thanks. I misunderstood something about those hooks apparently. I think this was originally inspired by migration logic in Cloud-Hypervisor, which freezes/unfreezes a partition during migration
- but only after the pre-copy phase is done.

I assumed the hooks were also called only before the final delta is transmitted. In tests I was relying on the reported state of the source from QMP, which probably can not accurately reflect the state of the partition as the freeze/unfreeze happens out of band (i.e. is is not orchestrated by the migration framework).

I'll have to research a bit and see if there are more appropriate hooks, or if we don't need to freeze the source partition at all.

best,

magnus

#39 lore raw

Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler

In reply to Paolo Bonzini
Quoted 10 lines · Paolo Bonzini

Sorry, I forgot to reply about this one.

It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while the VM runs, and the SaveVMHandler should just fail migration of the VM is running.

Paolo

Hey Paolo,

I spent some time looking at this. it's a bit twisted. The un/pause() calls are unfortunately named, they are merely freezing the progress of the reference timer, which for any real-world workload is in fact pausing the guest b/c if the timer doesn't progress an os would mostly stall, waiting for ticks.

We want to freeze the timer though during the last phase of the migration on the source and on the destination (can't set the reference time if the vcpu time is not frozen). But currently it's the wrong place. I'll find some more fitting place to perform this ceremony.

Curiously the time-frozen source was obscuring another potentially more serious bug. The mshv vcpu dispatch loop currently does not honor a cpu->exit_request, like it is currently set after precopy:

migration_completion_precopy()
migration_stop_vm()

    vm_stop_force_state()
      vm_stop()
        do_vm_stop()
          pause_all_vcpus()
            cpu_pause()
              cpu_exit()
                qatomic_store_release(&cpu->exit_request, true)

The two bugs were sort of cancelling each other out. Since the source guest was not busy, migrations were still succeeding, even though the the vcpus were not actually "stopped" on the vcpu.

I'll rework this a bit, and submit a revised version.

best,

magnus

maintainer
#40 lore raw

Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler

Il ven 3 lug 2026, 19:22 Magnus Kulke <[email protected]> ha scritto:

Quoted 13 lines · Paolo Bonzini
Quoted 5 lines

Sorry, I forgot to reply about this one.

It is not clear why you need to pause, since this is handled by common code. Are you not handling live migration yet? That should be handled at levels above QEMU, because users expect migration to be performed while

the

Quoted 5 lines

VM runs, and the SaveVMHandler should just fail migration of the VM is running.

Paolo

Quoted 8 lines

Hey Paolo,

I spent some time looking at this. it's a bit twisted. The un/pause() calls are unfortunately named, they are merely freezing the progress of the reference timer, which for any real-world workload is in fact pausing the guest b/c if the timer doesn't progress an os would mostly stall, waiting for ticks.

There are different "schools of thought" for that. Freezing introduces an incorrect timing on the destination, not freezing introduces a jump.

If you choose to freeze that's fine, QEMU does the same for KVM even though KVM has alternative ioctls that allow not freezing.

But currently it's the wrong

Quoted 2 lines

place. I'll find some more fitting place to perform this ceremony.

KVM uses a device for that, see kvnclock.c.

Curiously the time-frozen source was obscuring another potentially more

Quoted 2 lines

serious bug. The mshv vcpu dispatch loop currently does not honor a cpu->exit_request

Was that related to the other patch that was touching the vcpu thread loop?

Paolo

like it is currently set after precopy:

Quoted 22 lines

migration_completion_precopy()
migration_stop_vm()

    vm_stop_force_state()
      vm_stop()
        do_vm_stop()
          pause_all_vcpus()
            cpu_pause()
              cpu_exit()
                qatomic_store_release(&cpu->exit_request, true)

The two bugs were sort of cancelling each other out. Since the source guest was not busy, migrations were still succeeding, even though the the vcpus were not actually "stopped" on the vcpu.

I'll rework this a bit, and submit a revised version.

best,

magnus

#41 lore raw

Re: [PATCH 26/34] accel/mshv: introduce SaveVMHandler

Quoted 11 lines · Paolo Bonzini

There are different "schools of thought" for that. Freezing introduces an incorrect timing on the destination, not freezing introduces a jump.

If you choose to freeze that's fine, QEMU does the same for KVM even though KVM has alternative ioctls that allow not freezing.

But currently it's the wrong

Quoted 2 lines

place. I'll find some more fitting place to perform this ceremony.

KVM uses a device for that, see kvnclock.c.

Makes sense, kvmclock looks like a good template. I'll use a similar object.

Quoted 8 lines

Curiously the time-frozen source was obscuring another potentially more

Quoted 2 lines

serious bug. The mshv vcpu dispatch loop currently does not honor a cpu->exit_request

Was that related to the other patch that was touching the vcpu thread loop?

yup, we need the same here, I'll address it as part of a migration-pt2 series. we only need to check for cpu->exit_request and break the loop if set.

best,

magnus

#42 lore raw

PATCH 27/34 accel/mshv: write synthetic MSRs after migration

In reply to Magnus Kulke

Write partition-wide synthetic MSRs. This ensures the hypercall page and SynIC facilities are set up before vCPUs attempt to use it.

---
 accel/mshv/mshv-all.c          |  7 +++++++
 include/hw/hyperv/hvgdk_mini.h |  3 +++
 include/system/mshv_int.h      |  1 +
 target/i386/mshv/mshv-cpu.c    | 15 +++++++++++++++
 4 files changed, 26 insertions(+)
accel/mshv/mshv-all.c +7 -0
index 5f81881c05..e97f480551 100644
@@ -137,6 +137,13 @@ static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)
 static int mshv_load_cleanup(void *opaque)
 {
     MshvState *s = opaque;
+    int ret;
+
+    ret = mshv_arch_set_partition_msrs(first_cpu);
+    if (ret < 0) {
+        error_report("Failed to set partition MSRs: %s", strerror(-ret));
+        return -1;
+    }
 
     resume_vm(s->vm);
 
include/hw/hyperv/hvgdk_mini.h +3 -0
index 75e953b86c..e987f59bb9 100644
@@ -23,6 +23,9 @@
 #define HV_X64_MSR_APIC_FREQUENCY   0x40000023
 
 typedef enum hv_register_name {
+    /* VP Management Registers */
+    HV_REGISTER_INTERNAL_ACTIVITY_STATE = 0x00000004,
+
     /* Pending Interruption Register */
     HV_REGISTER_PENDING_INTERRUPTION = 0x00010002,
 
include/system/mshv_int.h +1 -0
index 7d685fc647..7052f20a00 100644
@@ -84,6 +84,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,
                           size_t n_regs);
 int mshv_arch_store_vcpu_state(const CPUState *cpu);
 int mshv_arch_load_vcpu_state(CPUState *cpu);
+int mshv_arch_set_partition_msrs(const CPUState *cpu);
 void mshv_arch_init_vcpu(CPUState *cpu);
 void mshv_arch_destroy_vcpu(CPUState *cpu);
 void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +15 -0
index 9ce66e9aa1..43dbbd6fbd 100644
@@ -1115,6 +1115,21 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)
     return 0;
 }
 
+int mshv_arch_set_partition_msrs(const CPUState *cpu)
+{
+    CPUX86State *env = &X86_CPU(cpu)->env;
+    struct hv_register_assoc assocs[] = {
+        { .name = HV_REGISTER_GUEST_OS_ID,
+          .value.reg64 = env->msr_hv_guest_os_id },
+        { .name = HV_REGISTER_REFERENCE_TSC,
+          .value.reg64 = env->msr_hv_tsc },
+        { .name = HV_X64_REGISTER_HYPERCALL,
+          .value.reg64 = env->msr_hv_hypercall },
+    };
+
+    return mshv_set_generic_regs(cpu, assocs, ARRAY_SIZE(assocs));
+}
+
 void mshv_arch_amend_proc_features(
     union hv_partition_synthetic_processor_features *features)
 {
Signature · 1 line
2.34.1
#43 lore raw

PATCH 28/34 accel/mshv: migrate REFERENCE_TIME

In reply to Magnus Kulke

This is a partition-wide state that we use a dedicated handler for. It is stored as-is in the "mshv" savevm_live field. We might have to extend this later if we have more partition-wide state to roundtrip.

---
 accel/mshv/mshv-all.c | 70 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 69 insertions(+), 1 deletion(-)
accel/mshv/mshv-all.c +69 -1
index e97f480551..facdc733e7 100644
@@ -85,13 +85,81 @@ static int set_time_freeze(int vm_fd, int freeze)
     return 0;
 }
 
+static int get_reference_time(int vm_fd, uint64_t *ref_time)
+{
+    int ret;
+    struct hv_input_get_partition_property in = {0};
+    struct hv_output_get_partition_property out = {0};
+    struct mshv_root_hvcall args = {0};
+
+    in.property_code = HV_PARTITION_PROPERTY_REFERENCE_TIME;
+
+    args.code = HVCALL_GET_PARTITION_PROPERTY;
+    args.in_sz = sizeof(in);
+    args.in_ptr = (uint64_t)&in;
+    args.out_sz = sizeof(out);
+    args.out_ptr = (uint64_t)&out;
+
+    ret = mshv_hvcall(vm_fd, &args);
+    if (ret < 0) {
+        error_report("Failed to get reference time");
+        return -1;
+    }
+
+    *ref_time = out.property_value;
+    return 0;
+}
+
 static void mshv_save_state(QEMUFile *f, void *opaque)
 {
-    return;
+    MshvState *s = opaque;
+    uint64_t ref_time;
+    int ret;
+
+    ret = get_reference_time(s->vm, &ref_time);
+    if (ret < 0) {
+        error_report("Failed to get reference time for migration");
+        abort();
+    }
+
+    qemu_put_be64(f, ref_time);
+}
+
+static int set_reference_time(int vm_fd, uint64_t ref_time)
+{
+    int ret;
+    struct hv_input_set_partition_property in = {0};
+    struct mshv_root_hvcall args = {0};
+
+    in.property_code = HV_PARTITION_PROPERTY_REFERENCE_TIME;
+    in.property_value = ref_time;
+
+    args.code = HVCALL_SET_PARTITION_PROPERTY;
+    args.in_sz = sizeof(in);
+    args.in_ptr = (uint64_t)&in;
+
+    ret = mshv_hvcall(vm_fd, &args);
+    if (ret < 0) {
+        error_report("Failed to set reference time");
+        return -1;
+    }
+
+    return 0;
 }
 
 static int mshv_load_state(QEMUFile *f, void *opaque, int version_id)
 {
+    MshvState *s = opaque;
+    uint64_t ref_time;
+    int ret;
+
+    ref_time = qemu_get_be64(f);
+
+    ret = set_reference_time(s->vm, ref_time);
+    if (ret < 0) {
+        error_report("Failed to set reference time after migration");
+        return -1;
+    }
     return 0;
 }
 
Signature · 1 line
2.34.1
#44 lore raw

PATCH 29/34 target/i386/mshv: migrate pending ints/excs

In reply to Magnus Kulke

We use PENDING_INTERRUPTION, INTERRUPT_STATE, PENDING_EVENT hv registers to map and roundtrip from/to CPUX86State.

We ignore HV_REGISTER_PENDING_EVENT1 which represent events for nested virt contexts, as we don't support nested virt with MSHV currently.

---
 include/hw/hyperv/hvgdk_mini.h |   3 +
 include/system/mshv_int.h      |  13 +++
 target/i386/mshv/mshv-cpu.c    | 168 +++++++++++++++++++++++++++++++++
 3 files changed, 184 insertions(+)
include/hw/hyperv/hvgdk_mini.h +3 -0
index e987f59bb9..e3fd2530ac 100644
@@ -28,6 +28,9 @@ typedef enum hv_register_name {
 
     /* Pending Interruption Register */
     HV_REGISTER_PENDING_INTERRUPTION = 0x00010002,
+    HV_REGISTER_INTERRUPT_STATE      = 0x00010003,
+    HV_REGISTER_PENDING_EVENT0       = 0x00010004,
+    HV_REGISTER_PENDING_EVENT1       = 0x00010005,
 
     /* X64 User-Mode Registers */
     HV_X64_REGISTER_RAX     = 0x00020000,
include/system/mshv_int.h +13 -0
index 7052f20a00..bc16b794b2 100644
@@ -18,6 +18,19 @@
 
 struct mshv_get_set_vp_state;
 
+/*
+ * Interruption-type encoding, used by the hypervisor in
+ * hv_x64_pending_interruption_register.interruption_type
+ * See TLFS 6.0 section 7.9.2, p55
+ * https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/tlfs
+ */
+#define MSHV_HV_INTERRUPTION_TYPE_EXT_INT     0
+#define MSHV_HV_INTERRUPTION_TYPE_NMI         2
+#define MSHV_HV_INTERRUPTION_TYPE_HW_EXC      3
+#define MSHV_HV_INTERRUPTION_TYPE_SW_INT      4
+#define MSHV_HV_INTERRUPTION_TYPE_PRIV_SW_EXC 5
+#define MSHV_HV_INTERRUPTION_TYPE_SW_EXC      6
+
 typedef struct hyperv_message hv_message;
 
 typedef struct MshvHvCallArgs {
target/i386/mshv/mshv-cpu.c +168 -0

186 lines not on this page · raw message

Signature · 1 line
2.34.1
#45 lore raw

PATCH 30/34 target/i386: add de/compaction to xsave_helper

In reply to Magnus Kulke

HyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.

- decompact_xsave_area(): converts compacted format to standard. XSTATE_BV is masked to host XCR0 since IA32_XSS is managed by the hypervisor.

- compact_xsave_area(): converts standard format back to compacted format. XCOMP_BV is set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.

Both functions use the host's CPUID leaf 0xD subleaves to determine component sizes, offsets, and alignment requirements.

There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (x86_cpu_expand_features).

---
 target/i386/cpu.h          |   2 +
 target/i386/xsave_helper.c | 255 +++++++++++++++++++++++++++++++++++++
 2 files changed, 257 insertions(+)
target/i386/cpu.h +2 -0
index 4ad4a35ce9..cd5d5a5369 100644
@@ -3033,6 +3033,8 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen);
 void x86_cpu_xsave_all_areas(X86CPU *cpu, void *buf, uint32_t buflen);
 uint32_t xsave_area_size(uint64_t mask, bool compacted);
 void x86_update_hflags(CPUX86State* env);
+int decompact_xsave_area(const void *buf, size_t buflen, CPUX86State *env);
+int compact_xsave_area(CPUX86State *env, void *buf, size_t buflen);
 
 static inline bool hyperv_feat_enabled(X86CPU *cpu, int feat)
 {
target/i386/xsave_helper.c +255 -0

264 lines not on this page · raw message

Signature · 1 line
2.34.1
Reviewed-by on v1 30/34
#46 lore raw

Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper

Quoted 26 lines · Magnus Kulke

HyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.

- decompact_xsave_area(): converts compacted format to standard. XSTATE_BV is masked to host XCR0 since IA32_XSS is managed by the hypervisor.

- compact_xsave_area(): converts standard format back to compacted format. XCOMP_BV is set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.

Both functions use the host's CPUID leaf 0xD subleaves to determine component sizes, offsets, and alignment requirements.

There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (x86_cpu_expand_features).

Quoted 5 lines

---
target/i386/cpu.h | 2 +
target/i386/xsave_helper.c | 255 +++++++++++++++++++++++++++++++++++++

2 files changed, 257 insertions(+)
Quoted 288 lines
target/i386/cpu.h +2 -0
index 4ad4a35ce9..cd5d5a5369 100644
@@ -3033,6 +3033,8 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen);
 void x86_cpu_xsave_all_areas(X86CPU *cpu, void *buf, uint32_t buflen);
 uint32_t xsave_area_size(uint64_t mask, bool compacted);
 void x86_update_hflags(CPUX86State* env);
+int decompact_xsave_area(const void *buf, size_t buflen, CPUX86State *env);
+int compact_xsave_area(CPUX86State *env, void *buf, size_t buflen);
 
 static inline bool hyperv_feat_enabled(X86CPU *cpu, int feat)
 {
target/i386/xsave_helper.c +255 -0
index bab2258732..2272b83f5f 100644
@@ -3,6 +3,7 @@
 * See the COPYING file in the top-level directory.
 */
 #include "qemu/osdep.h"
+#include "qemu/error-report.h"
 
 #include "cpu.h"
 
@@ -293,3 +294,257 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen)
    }
 #endif
 }
+
+#define XSTATE_BV_IN_HDR  offsetof(X86XSaveHeader, xstate_bv)
+#define XCOMP_BV_IN_HDR   offsetof(X86XSaveHeader, xcomp_bvo)
+
+typedef struct X86XSaveAreaView {
+    /* 512 bytes */
+    X86LegacyXSaveArea legacy;
+    /* 64 bytes */
+    X86XSaveHeader     header;
+    /* ...followed by individual xsave areas */
+} X86XSaveAreaView;
+
+#define XSAVE_XSTATE_BV_OFFSET  offsetof(X86XSaveAreaView, header.xstate_bv)
+#define XSAVE_XCOMP_BV_OFFSET   offsetof(X86XSaveAreaView, header.xcomp_bv)
+#define XSAVE_EXT_OFFSET        (sizeof(X86LegacyXSaveArea) + \
+                                 sizeof(X86XSaveHeader))
+
+/**
+ * decompact_xsave_area - Convert compacted XSAVE format to standard format
+ * @buf: Source buffer containing compacted XSAVE data
+ * @buflen: Size of source buffer
+ * @env: CPU state where the standard format buffer will be written to
+ *
+ * Accelerator backends like MSHV might return XSAVE state in compacted format
+ * (XSAVEC). The state components have to be packed contiguously without gaps.
+ * The XSAVE qemu buffers are in standard format where each component has a
+ * fixed offset.
+ *
+ * Returns: 0 on success, negative errno on failure
+ */
+int decompact_xsave_area(const void *buf, size_t buflen, CPUX86State *env)
+{
+    uint64_t compacted_xstate_bv, compacted_xcomp_bv, compacted_layout_bv;
+    uint64_t xsave_offset, *xcomp_bv;
+    size_t i;
+    uint32_t eax, ebx, ecx, edx;
+    uint32_t size, dst_off;
+    bool align64;
+    uint64_t guest_xcr0, *xstate_bv;
+
+    compacted_xstate_bv = *(uint64_t *)(buf + XSAVE_XSTATE_BV_OFFSET);
+    compacted_xcomp_bv  = *(uint64_t *)(buf + XSAVE_XCOMP_BV_OFFSET);
+
+    /* This function only handles compacted format (bit 63 set) */
+    assert((compacted_xcomp_bv >> 63) & 1);
+
+    /* Low bits of XCOMP_BV describe which components are in the layout */
+    compacted_layout_bv = compacted_xcomp_bv & ~(1ULL << 63);
+
+    /* Zero out buffer, then copy legacy region (FP + SSE) and header as-is */
+    memset(env->xsave_buf, 0, env->xsave_buf_len);
+    memcpy(env->xsave_buf, buf, XSAVE_EXT_OFFSET);
+
+    /*
+     * We mask XSTATE_BV with the guest's supported XCR0 because:
+     * 1. Supervisor state (IA32_XSS) is hypervisor-managed, we don't use
+     *    this state for migration.
+     * 2. Features disabled at partition creation (e.g. AMX) must be excluded
+     */
+    guest_xcr0 = ((uint64_t)env->features[FEAT_XSAVE_XCR0_HI] << 32) |
+                 env->features[FEAT_XSAVE_XCR0_LO];
+    xstate_bv = (uint64_t *)(env->xsave_buf + XSAVE_XSTATE_BV_OFFSET);
+    *xstate_bv &= guest_xcr0;
+
+    /* Clear bit 63 - output is standard format, not compacted */
+    xcomp_bv = (uint64_t *)(env->xsave_buf + XSAVE_XCOMP_BV_OFFSET);
+    *xcomp_bv = *xcomp_bv & ~(1ULL << 63);
+
+    /*
+     * Process each extended state component in the compacted layout.
+     * Components 0 and 1 (FP and SSE) are in the legacy region, so we
+     * start at component 2. For each component:
+     * - Calculate its offset in the compacted source (contiguous layout)
+     * - Get its fixed offset in the standard destination from CPUID
+     * - Copy if the component has non-init state (bit set in XSTATE_BV)
+     */
+    xsave_offset = XSAVE_EXT_OFFSET;
+    for (i = 2; i < 63; i++) {
+        if (((compacted_layout_bv >> i) & 1) == 0) {
+            continue;
+        }
+
+        /* Query guest CPUID for this component's size and standard offset */
+        cpu_x86_cpuid(env, 0xD, i, &eax, &ebx, &ecx, &edx);
+
+        size = eax;
+        dst_off = ebx;
+        align64 = (ecx & (1u << 1)) != 0;
+
+        /* Component is in the layout but unknown to the guest CPUID model */
+        if (size == 0) {
+            /*
+             * The hypervisor might expose a component that has no
+             * representation in the guest CPUID model. We query the host to
+             * retrieve the size of the component, so we can skip over it.
+             */
+            host_cpuid(0xD, i, &eax, &ebx, &ecx, &edx);
+            size = eax;
+            align64 = (ecx & (1u << 1)) != 0;
+            if (size == 0) {
+                error_report("xsave component %zu: size unknown to both "
+                             "guest and host CPUID", i);
+                return -EINVAL;
+            }
+
+            if (align64) {
+                xsave_offset = QEMU_ALIGN_UP(xsave_offset, 64);
+            }
+
+            if (xsave_offset + size > buflen) {
+                error_report("xsave component %zu overruns source buffer: "
+                             "offset=%zu size=%u buflen=%zu",
+                             i, xsave_offset, size, buflen);
+                return -E2BIG;
+            }
+
+            xsave_offset += size;
+            continue;
+        }
+
+        if (align64) {
+            xsave_offset = QEMU_ALIGN_UP(xsave_offset, 64);
+        }
+
+        if ((xsave_offset + size) > buflen) {
+            error_report("xsave component %zu overruns source buffer: "
+                         "offset=%zu size=%u buflen=%zu",
+                         i, xsave_offset, size, buflen);
+            return -E2BIG;
+        }
+
+        if ((dst_off + size) > env->xsave_buf_len) {
+            error_report("xsave component %zu overruns destination buffer: "
+                         "offset=%u size=%u buflen=%zu",
+                         i, dst_off, size, (size_t)env->xsave_buf_len);
+            return -E2BIG;
+        }
+
+        /* Copy components marked present in XSTATE_BV to guest model */
+        if (((compacted_xstate_bv >> i) & 1) != 0) {
+            memcpy(env->xsave_buf + dst_off, buf + xsave_offset, size);
+        }
+
+        xsave_offset += size;
+    }
+
+    return 0;
+}
+
+/**
+ * compact_xsave_area - Convert standard XSAVE format to compacted format
+ * @env: CPU state containing the standard format XSAVE buffer
+ * @buf: Destination buffer for compacted XSAVE data (to send to hypervisor)
+ * @buflen: Size of destination buffer
+ *
+ * Accelerator backends like MSHV might expect XSAVE state in compacted format
+ * (XSAVEC). The state components are packed contiguously without gaps.
+ * The XSAVE qemu buffers are in standard format where each component has a
+ * fixed offset.
+ *
+ * This function converts from standard to compacted format, it accepts a
+ * pre-allocated destination buffer of sufficient size, it is the
+ * responsibility of the caller to ensure the buffer is big enough.
+ *
+ * Returns: total size of compacted XSAVE data written to @buf
+ */
+int compact_xsave_area(CPUX86State *env, void *buf, size_t buflen)
+{
+    uint64_t *xcomp_bv;
+    size_t i;
+    uint32_t eax, ebx, ecx, edx;
+    uint32_t size, src_off;
+    bool align64;
+    size_t compact_offset;
+    uint64_t host_xcr0_mask, guest_xcr0;
+
+    /* Zero out buffer, then copy legacy region (FP + SSE) and header as-is */
+    memset(buf, 0, buflen);
+    memcpy(buf, env->xsave_buf, XSAVE_EXT_OFFSET);
+
+    /*
+     * Set XCOMP_BV to indicate compacted format (bit 63) and which
+     * components are in the layout.
+     *
+     * We must explicitly set XCOMP_BV because x86_cpu_xsave_all_areas()
+     * produces standard format with XCOMP_BV=0 (buffer is zeroed and only
+     * XSTATE_BV is set in the header).
+     *
+     * XCOMP_BV must reflect the partition's XSAVE capability, not the
+     * guest's current XCR0 (env->xcr0). These differ b/c:
+     * - A guest's XCR0 is what the guest OS has enabled via XSETBV
+     * - The partition's XCR0 mask is the hypervisor's save/restore capability
+     *
+     * The hypervisor uses XSAVES which saves based on its capability, so the
+     * XCOMP_BV value in the buffer we send back must match that capability.
+     *
+     * We intersect the host XCR0 with the guest's supported XCR0 features
+     * (FEAT_XSAVE_XCR0_*) so that features disabled at partition creation
+     * (e.g. AMX) are excluded from the compacted layout.
+     */
+    host_cpuid(0xD, 0, &eax, &ebx, &ecx, &edx);
+    host_xcr0_mask = ((uint64_t)edx << 32) | eax;
+    guest_xcr0 = ((uint64_t)env->features[FEAT_XSAVE_XCR0_HI] << 32) |
+                 env->features[FEAT_XSAVE_XCR0_LO];
+    host_xcr0_mask &= guest_xcr0;
+    xcomp_bv = buf + XSAVE_XCOMP_BV_OFFSET;
+    *xcomp_bv = host_xcr0_mask | (1ULL << 63);
+
+    /*
+     * Process each extended state component in the host's XCR0.
+     * The compacted layout must match XCOMP_BV (host capability).
+     *
+     * For each component:
+     * - Get its size and standard offset from host CPUID
+     * - Apply 64-byte alignment if required
+     * - Copy data only if guest has this component (bit set in env->xcr0)
+     * - Always advance offset to maintain correct layout
+     */
+    compact_offset = XSAVE_EXT_OFFSET;
+    for (i = 2; i < 63; i++) {
+        if (!((host_xcr0_mask >> i) & 1)) {
+            continue;
+        }
+
+        /* Query host CPUID for this component's size and standard offset */
+        host_cpuid(0xD, i, &eax, &ebx, &ecx, &edx);
+        size = eax;
+        src_off = ebx;
+        align64 = (ecx >> 1) & 1;
+
+        if (size == 0) {
+            /* Component in host xcr0 but unknown - shouldn't happen */
+            continue;
+        }
+
+        /* Apply 64-byte alignment if required by this component */
+        if (align64) {
+            compact_offset = QEMU_ALIGN_UP(compact_offset, 64);
+        }
+
+        /*
+         * Only copy data if guest has this component enabled in XCR0.
+         * Otherwise the component remains zeroed (init state), but we
+         * still advance the offset to maintain the correct layout.
+         */
+        if ((env->xcr0 >> i) & 1) {
+            memcpy(buf + compact_offset, env->xsave_buf + src_off, size);
+        }
+
+        compact_offset += size;
+    }
+
+    return compact_offset;
+}
Signature · 1 line
2.34.1
#47 lore raw

Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper

In reply to Magnus Kulke
Quoted 21 lines · Magnus Kulke

HyperV use XSAVES which stores extended state in compacted format in which components are packed contiguously, while QEMU's internal XSAVE representation use the standard format in which each component is places at a fixed offset. Hence for this purpose we add two conversion fn's to the xsave helper to roundtrip XSAVE state in a migration.

- decompact_xsave_area(): converts compacted format to standard. XSTATE_BV is masked to host XCR0 since IA32_XSS is managed by the hypervisor.

- compact_xsave_area(): converts standard format back to compacted format. XCOMP_BV is set from the host's CPUID 0xD.0 rather than the guest's XCR0, as this is what the hypervisor expects.

Both functions use the host's CPUID leaf 0xD subleaves to determine component sizes, offsets, and alignment requirements.

Quoted 6 lines

There are situations when the host advertises features that we want to disable for the guest, e.g. AMX TILE. In this case we cannot rely on the host's xcr0, but instead we use the feature mask that has been generated in as part of the CPU realization process (x86_cpu_expand_features).

Hi,

This patch has a tiny problem.

#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
void *xsave_buf;
uint32_t xsave_buf_len;
#endif

With the "#if defined(CONFIG_KVM) || defined(CONFIG_HVF)” removed, that fixes a TCG only QEMU build.

Was also able to test that this works as expected on WHPX.

#48 lore raw

Re: [PATCH 30/34] target/i386: add de/compaction to xsave_helper

Quoted 10 lines · Mohamed Mediouni
Quoted 3 lines · Magnus Kulke

This patch has a tiny problem.

#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
void *xsave_buf;
uint32_t xsave_buf_len;
#endif

Quoted 4 lines

With the "#if defined(CONFIG_KVM) || defined(CONFIG_HVF)” removed, that fixes a TCG only QEMU build.

Was also able to test that this works as expected on WHPX.

thx, that's right. we can also change the signature to pass xsave_buf and xsave_buf_len as params, so accelerators which do not have xsave buffers do not choke on this, it's probably less invasive rather than including xsave_buf fields for all accelerators.

#49 lore raw

PATCH 31/34 target/i386/mshv: migrate XSAVE state

In reply to Magnus Kulke

We implement fn's that roundtrip XSAVE state in migration. We are using the xsave_helper routines to move individual components from CPUX86State to an xsave_buf and then we have to compact the buffer to XSAVEC format, which is what the hypervisor expects.

And the same applies in the other direction for restoring state from the hypervisor.

---
 target/i386/cpu.h           |   2 +-
 target/i386/mshv/mshv-cpu.c | 100 +++++++++++++++++++++++++++++++++++-
 2 files changed, 100 insertions(+), 2 deletions(-)
target/i386/cpu.h +1 -1
index cd5d5a5369..0f30f0dd5b 100644
@@ -2272,7 +2272,7 @@ typedef struct CPUArchState {
     int64_t user_tsc_khz; /* for sanity check only */
     uint64_t apic_bus_freq;
     uint64_t tsc;
-#if defined(CONFIG_KVM) || defined(CONFIG_HVF)
+#if defined(CONFIG_KVM) || defined(CONFIG_HVF) || defined(CONFIG_MSHV)
     void *xsave_buf;
     uint32_t xsave_buf_len;
 #endif
target/i386/mshv/mshv-cpu.c +99 -1

138 lines not on this page · raw message

Signature · 1 line
2.34.1
#50 lore raw

PATCH 32/34 target/i386/mshv: reconstruct hflags after load

In reply to Magnus Kulke

hflags is a cached bitmap derived from standard and special regs. We want to reconstruct this state after regs and sregs have been read from the hypervisor, similar to how it's one in other accelerators.

---
 target/i386/mshv/mshv-cpu.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
target/i386/mshv/mshv-cpu.c +13 -0
index 5b0b8bda16..463bf7efa0 100644
@@ -814,6 +814,16 @@ static int set_vcpu_events(const CPUState *cpu)
     return 0;
 }
 
+static int update_hflags(CPUState *cpu)
+{
+    X86CPU *x86cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86cpu->env;
+
+    x86_update_hflags(env);
+
+    return 0;
+}
+
 int mshv_arch_load_vcpu_state(CPUState *cpu)
 {
     int ret;
@@ -828,6 +838,9 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)
         return ret;
     }
 
+    /* INVARIANT: hflags are derived from regs+sregs, need to get both first */
+    update_hflags(cpu);
+
     ret = get_xc_reg(cpu);
     if (ret < 0) {
         return ret;
Signature · 1 line
2.34.1
#51 lore raw

PATCH 33/34 target/i386/mshv: migrate MP_STATE

In reply to Magnus Kulke

MSHV's "internal activity state" roughly maps to QEMU's env->mp_state and cpu->halted states that describe state of APs in a guest.

We don't invoke set_mp_state as part of store_vcpu_state() b/c we would put all BSP + APs in a RUNNABLE (0) state immediately, breaking SMP boot

Instead we store the mp state as part of the load_cleanup() routine after a migration.

---
 accel/mshv/mshv-all.c       | 10 +++++
 include/system/mshv_int.h   |  1 +
 target/i386/mshv/mshv-cpu.c | 80 +++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+)
accel/mshv/mshv-all.c +10 -0
index facdc733e7..ffe84d6151 100644
@@ -205,6 +205,7 @@ static int mshv_load_setup(QEMUFile *f, void *opaque, Error **errp)
 static int mshv_load_cleanup(void *opaque)
 {
     MshvState *s = opaque;
+    CPUState *cpu;
     int ret;
 
     ret = mshv_arch_set_partition_msrs(first_cpu);
@@ -213,6 +214,15 @@ static int mshv_load_cleanup(void *opaque)
         return -1;
     }
 
+    CPU_FOREACH(cpu) {
+        ret = mshv_arch_set_mp_state(cpu);
+        if (ret < 0) {
+            error_report("Failed to set mp state for vCPU %d: %s",
+                         cpu->cpu_index, strerror(-ret));
+            return -1;
+        }
+    }
+
     resume_vm(s->vm);
 
     return 0;
include/system/mshv_int.h +1 -0
index bc16b794b2..c24efc8675 100644
@@ -98,6 +98,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,
 int mshv_arch_store_vcpu_state(const CPUState *cpu);
 int mshv_arch_load_vcpu_state(CPUState *cpu);
 int mshv_arch_set_partition_msrs(const CPUState *cpu);
+int mshv_arch_set_mp_state(const CPUState *cpu);
 void mshv_arch_init_vcpu(CPUState *cpu);
 void mshv_arch_destroy_vcpu(CPUState *cpu);
 void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +80 -0

98 lines not on this page · raw message

Signature · 1 line
2.34.1
#52 lore raw

PATCH 34/34 accel/mshv: enable dirty page tracking

In reply to Magnus Kulke

This change introduces the functions required to perform dirty page tracking to speed up migrations. We are using the sync, global_start, and global_stop hooks.

The sync is implemented in batches.

Before we can disable the dirty page tracking we have to set all dirty bits.

---
 accel/mshv/mem.c          | 211 ++++++++++++++++++++++++++++++++++++++
 accel/mshv/mshv-all.c     |   3 +
 include/system/mshv_int.h |   5 +
 3 files changed, 219 insertions(+)
accel/mshv/mem.c +211 -0
index e55c38d4db..820f87ef0c 100644
@@ -12,10 +12,13 @@
 
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
+#include "qapi/error.h"
 #include "linux/mshv.h"
 #include "system/address-spaces.h"
 #include "system/mshv.h"
 #include "system/mshv_int.h"
+#include "hw/hyperv/hvhdk_mini.h"
+#include "system/physmem.h"
 #include "exec/memattrs.h"
 #include <sys/ioctl.h>
 #include "trace.h"
@@ -211,3 +214,211 @@ void mshv_set_phys_mem(MshvMemoryListener *mml, MemoryRegionSection *section,
         abort();
     }
 }
+
+static int enable_dirty_page_tracking(int vm_fd)
+{
+    int ret;
+    struct hv_input_set_partition_property in = {0};
+    struct mshv_root_hvcall args = {0};
+
+    in.property_code = HV_PARTITION_PROPERTY_GPA_PAGE_ACCESS_TRACKING;
+    in.property_value = 1;
+
+    args.code = HVCALL_SET_PARTITION_PROPERTY;
+    args.in_sz = sizeof(in);
+    args.in_ptr = (uint64_t)&in;
+
+    ret = mshv_hvcall(vm_fd, &args);
+    if (ret < 0) {
+        error_report("Failed to enable dirty page tracking: %s",
+                     strerror(errno));
+        return -1;
+    }
+
+    return 0;
+}
+
+/*
+ * Retrieve dirty page bitmap for a GPA range, clearing the dirty bits
+ * atomically. Large ranges are handled in batches.
+ */
+static int get_dirty_log(int vm_fd, uint64_t base_pfn, uint64_t page_count,
+                         unsigned long *bitmap, size_t bitmap_size)
+{
+    uint64_t batch, bitmap_offset, completed = 0;
+    struct mshv_gpap_access_bitmap args = {0};
+    int ret;
+
+    QEMU_BUILD_BUG_ON(MSHV_DIRTY_PAGES_BATCH_SIZE % BITS_PER_LONG != 0);
+    assert(bitmap_size >= ROUND_UP(page_count, BITS_PER_LONG) / 8);
+
+    while (completed < page_count) {
+        batch = MIN(MSHV_DIRTY_PAGES_BATCH_SIZE, page_count - completed);
+        bitmap_offset = completed / BITS_PER_LONG;
+
+        args.access_type = MSHV_GPAP_ACCESS_TYPE_DIRTY;
+        args.access_op   = MSHV_GPAP_ACCESS_OP_CLEAR;
+        args.page_count  = batch;
+        args.gpap_base   = base_pfn + completed;
+        args.bitmap_ptr  = (uint64_t)(bitmap + bitmap_offset);
+
+        ret = ioctl(vm_fd, MSHV_GET_GPAP_ACCESS_BITMAP, &args);
+        if (ret < 0) {
+            error_report("Failed to get dirty log (base_pfn=0x%" PRIx64
+                         " batch=%" PRIu64 "): %s",
+                         base_pfn + completed, batch, strerror(errno));
+            return -1;
+        }
+        completed += batch;
+    }
+
+    return 0;
+}
+
+bool mshv_log_global_start(MemoryListener *listener, Error **errp)
+{
+    int ret;
+
+    ret = enable_dirty_page_tracking(mshv_state->vm);
+    if (ret < 0) {
+        error_setg_errno(errp, -ret, "Failed to enable dirty page tracking");
+        return false;
+    }
+    return true;
+}
+
+static int disable_dirty_page_tracking(int vm_fd)
+{
+    int ret;
+    struct hv_input_set_partition_property in = {0};
+    struct mshv_root_hvcall args = {0};
+
+    in.property_code = HV_PARTITION_PROPERTY_GPA_PAGE_ACCESS_TRACKING;
+    in.property_value = 0;
+
+    args.code = HVCALL_SET_PARTITION_PROPERTY;
+    args.in_sz = sizeof(in);
+    args.in_ptr = (uint64_t)&in;
+
+    ret = mshv_hvcall(vm_fd, &args);
+    if (ret < 0) {
+        error_report("Failed to disable dirty page tracking: %s",
+                     strerror(errno));
+        return -1;
+    }
+
+    return 0;
+}
+
+static int set_dirty_pages(int vm_fd, uint64_t base_pfn, uint64_t page_count)
+{
+    uint64_t batch, completed = 0;
+    unsigned long bitmap[MSHV_DIRTY_PAGES_BATCH_SIZE / BITS_PER_LONG];
+    struct mshv_gpap_access_bitmap args = {0};
+    int ret;
+
+    while (completed < page_count) {
+        batch = MIN(MSHV_DIRTY_PAGES_BATCH_SIZE, page_count - completed);
+
+        args.access_type = MSHV_GPAP_ACCESS_TYPE_DIRTY;
+        args.access_op   = MSHV_GPAP_ACCESS_OP_SET;
+        args.page_count  = batch;
+        args.gpap_base   = base_pfn + completed;
+        args.bitmap_ptr  = (uint64_t)bitmap;
+
+        ret = ioctl(vm_fd, MSHV_GET_GPAP_ACCESS_BITMAP, &args);
+        if (ret < 0) {
+            error_report("Failed to set dirty pages (base_pfn=0x%" PRIx64
+                         " batch=%" PRIu64 "): %s",
+                         base_pfn + completed, batch, strerror(errno));
+            return -1;
+        }
+        completed += batch;
+    }
+
+    return 0;
+}
+
+static bool set_dirty_bits_cb(Int128 start, Int128 len, const MemoryRegion *mr,
+                              hwaddr offset_in_region, void *opaque)
+{
+    int ret, *errp = opaque;
+    hwaddr gpa, size;
+    uint64_t page_count, base_pfn;
+
+    gpa = int128_get64(start);
+    size = int128_get64(len);
+    page_count = size >> MSHV_PAGE_SHIFT;
+    base_pfn = gpa >> MSHV_PAGE_SHIFT;
+
+    if (!mr->ram || mr->readonly) {
+        return false;
+    }
+
+    if (page_count == 0) {
+        return false;
+    }
+
+    ret = set_dirty_pages(mshv_state->vm, base_pfn, page_count);
+
+    /* true aborts the iteration, which is what we want if there's an error */
+    if (ret < 0) {
+        *errp = ret;
+        return true;
+    }
+
+    return false;
+}
+
+void mshv_log_global_stop(MemoryListener *listener)
+{
+    int err = 0;
+    /* MSHV requires all dirty bits to be set before disabling tracking. */
+    FlatView *fv = address_space_to_flatview(&address_space_memory);
+    flatview_for_each_range(fv, set_dirty_bits_cb, &err);
+
+    if (err < 0) {
+        error_report("Failed to set dirty bits before disabling tracking");
+    }
+
+    disable_dirty_page_tracking(mshv_state->vm);
+}
+
+void mshv_log_sync(MemoryListener *listener, MemoryRegionSection *section)
+{
+    hwaddr size, start_addr, mr_offset;
+    uint64_t page_count, base_pfn;
+    size_t bitmap_size;
+    unsigned long *bitmap;
+    ram_addr_t ram_addr;
+    int ret;
+    MemoryRegion *mr = section->mr;
+
+    if (!memory_region_is_ram(mr) || memory_region_is_rom(mr)) {
+        return;
+    }
+
+    size = align_section(section, &start_addr);
+    if (!size) {
+        return;
+    }
+
+    page_count = size >> MSHV_PAGE_SHIFT;
+    base_pfn = start_addr >> MSHV_PAGE_SHIFT;
+    bitmap_size = ROUND_UP(page_count, BITS_PER_LONG) / 8;
+    bitmap = g_malloc0(bitmap_size);
+
+    ret = get_dirty_log(mshv_state->vm, base_pfn, page_count, bitmap,
+                        bitmap_size);
+    if (ret < 0) {
+        g_free(bitmap);
+        return;
+    }
+
+    mr_offset = section->offset_within_region + start_addr -
+                section->offset_within_address_space;
+    ram_addr = memory_region_get_ram_addr(mr) + mr_offset;
+
+    physical_memory_set_dirty_lebitmap(bitmap, ram_addr, page_count);
+    g_free(bitmap);
+}
accel/mshv/mshv-all.c +3 -0
index ffe84d6151..94ff9cdb49 100644
@@ -546,6 +546,9 @@ static MemoryListener mshv_memory_listener = {
     .region_del = mem_region_del,
     .eventfd_add = mem_ioeventfd_add,
     .eventfd_del = mem_ioeventfd_del,
+    .log_sync = mshv_log_sync,
+    .log_global_start = mshv_log_global_start,
+    .log_global_stop = mshv_log_global_stop,
 };
 
 static MemoryListener mshv_io_listener = {
include/system/mshv_int.h +5 -0
index c24efc8675..ddbdd76076 100644
@@ -31,6 +31,8 @@ struct mshv_get_set_vp_state;
 #define MSHV_HV_INTERRUPTION_TYPE_PRIV_SW_EXC 5
 #define MSHV_HV_INTERRUPTION_TYPE_SW_EXC      6
 
+#define MSHV_DIRTY_PAGES_BATCH_SIZE 0x10000
+
 typedef struct hyperv_message hv_message;
 
 typedef struct MshvHvCallArgs {
@@ -128,6 +130,9 @@ int mshv_guest_mem_write(uint64_t gpa, const uint8_t *data, uintptr_t size,
                          bool is_secure_mode);
 void mshv_set_phys_mem(MshvMemoryListener *mml, MemoryRegionSection *section,
                        bool add);
+void mshv_log_sync(MemoryListener *listener, MemoryRegionSection *section);
+bool mshv_log_global_start(MemoryListener *listener, Error **errp);
+void mshv_log_global_stop(MemoryListener *listener);
 
 /* msr */
 int mshv_init_msrs(const CPUState *cpu);
Signature · 1 line
2.34.1
maintainer
#53 lore raw
In reply to Magnus Kulke

I merged everything except:

- target/i386/mshv: migrate LAPIC state (*)
- target/i386/mshv: migrate Synic SINT MSRs
- target/i386/mshv: migrate SIMP and SIEFP state
- target/i386/mshv: migrate STIMER state
- accel/mshv: introduce SaveVMHandler (*)
- accel/mshv: write synthetic MSRs after migration
- accel/mshv: migrate REFERENCE_TIME
- target/i386/mshv: migrate MP_STATE

The ones marked (*) were the important ones that I'll reply to, the rest are collateral damage.

I also removed TSC_DEADLINE from the MSRs patch, since it's more of a local APIC register and it has a dependency on "migrate LAPIC state".

Paolo