Add migration support to MSHV accelerator, Part 2
cover Add migration support to MSHV accelerator, Part 2
Hey all,
this is a follow up to a previous series "Add migration support to the MSHV accelerator".
There were 2 contested changes that have been reworked in this series
- target/i386/mshv: migrate LAPIC state
- accel/mshv: introduce SaveVMHandler
"Introduce SaveVMHandler" was adding lifecycle hooks that froze the timer at the start of the migration on the source, which effectively paused any workload that would depend on timer progress (like an os) during the migration window. This is undesired for live migration and has been split out into a dedicated clock device that has its own migration lifecycle in "hw/i386/mshv: migrate REFERENCE_TIME".
"Migrate LAPIC state" changed in 2 ways: 1) for mshv-specific fields there is now an "MshvAPICState" object that captures auxiliary fields that exist in the MSHV LAPIC state. Initially we also shadowed the lvt[APIC_LVT_LINT0/1] fields, since a write to those fields was triggering a userpace PIC delivery path that MSHV doesn't support.
2) We add a guard in "pic_irq_request()" to short-circuit the userland delivery for MSHV for PIC interrupts, that means can use the LINT0/1 fields from the CommonAPICState.
The other migration steps have been carried over more or less verbatim from the first patch series.
3 drive-by fixes:
- disable AMX TILE features: this is required on because we do not not support XSAVE buffers beyond 4k in the accelerator yet
- drop initial msi vector 0: this has produced a warning at launch, after tracing the source we can drop it silently with confidence.
- toggle fpu/xsave migration: a subtle bug where fpu state would be clobbered by xsave legacy space when applied in the wrong order.
best regards,
magnus
Magnus Kulke (12):
target/i386/mshv: disable AMX TILE features accel/mshv: introduce SaveVMHandler hw/i386/mshv: migrate REFERENCE_TIME accel/mshv: install dummy handler for SIG_IPI target/i386/mshv: migrate LAPIC state target/i386/mshv: migrate Synic SINT MSRs target/i386/mshv: migrate SIMP and SIEFP state target/i386/mshv: migrate STIMER state accel/mshv: write synthetic MSRs after migration target/i386/mshv: migrate MP_STATE target/i386/mshv: toggle fpu/xsave migration hw/i386/mshv: drop initial msi vector 0
MAINTAINERS | 1 + accel/mshv/mshv-all.c | 98 ++++---- accel/mshv/trace-events | 1 + hw/i386/meson.build | 1 + hw/i386/mshv/apic.c | 405 +++++++++++++++++++++++++++++++++ hw/i386/mshv/clock.c | 189 +++++++++++++++ hw/i386/mshv/meson.build | 5 + hw/i386/x86-cpu.c | 6 + include/hw/hyperv/hvgdk_mini.h | 5 + include/hw/i386/apic-msidef.h | 1 + include/system/mshv.h | 5 + include/system/mshv_int.h | 19 +- target/i386/cpu-apic.c | 3 + target/i386/cpu.h | 10 + target/i386/machine.c | 46 ++++ target/i386/mshv/meson.build | 1 + target/i386/mshv/mshv-cpu.c | 305 +++++++++++++++++-------- target/i386/mshv/msr.c | 40 ++++ target/i386/mshv/synic.c | 206 +++++++++++++++++ 19 files changed, 1196 insertions(+), 151 deletions(-) create mode 100644 hw/i386/mshv/apic.c create mode 100644 hw/i386/mshv/clock.c create mode 100644 hw/i386/mshv/meson.build create mode 100644 target/i386/mshv/synic.c
Signature · 1 line
2.34.1
PATCH 1/12 target/i386/mshv: disable AMX TILE features
For the time being we disable AMX TILE in partition processor features and CPUID b/c AMX TILE XSAVE state (XTILE_DATA) is 8KB, which exceeds the current fixed 4KB XSAVE buffer size.
For now we filter it until buffer sizing is computed dynamically from CPUID.
--- accel/mshv/mshv-all.c | 8 ++++++++ target/i386/mshv/mshv-cpu.c | 16 ++++++++++++++++ 2 files changed, 24 insertions(+)
accel/mshv/mshv-all.c +8 -0
@@ -203,6 +203,14 @@ static int create_partition(int mshv_fd, int *vm_fd)/* enable all */ disabled_xsave_features.as_uint64 =0 ; +/* +* AMX TILE XSAVE state (XTILE_DATA) is 8KB, which exceeds the +* current fixed 4KB XSAVE buffer size. +*/ + disabled_xsave_features.amx_tile_support =1 ; + disabled_xsave_features.amx_bf16_support =1 ; + disabled_xsave_features.amx_int8_support =1 ; + disabled_xsave_features.amx_fp16_support =1 ;/* * query host for supported processor features and disable unsupported
target/i386/mshv/mshv-cpu.c +16 -0
@@ -2156,6 +2156,22 @@ uint32_t mshv_get_supported_cpuid(uint32_t func, uint32_t idx, int reg)*/ ret &= ~CPUID_7_0_ECX_LA57 ; } +if (func ==0x07 && idx ==0 && reg ==R_EDX ) { +/* +* AMX TILE XSAVE state (XTILE_DATA) is 8KB, which exceeds the +* current fixed 4KB XSAVE buffer size. Filter until buffer +* sizing is computed dynamically from CPUID. +*/ + ret &= ~CPUID_7_0_EDX_AMX_TILE ; + ret &= ~CPUID_7_0_EDX_AMX_BF16 ; + ret &= ~CPUID_7_0_EDX_AMX_INT8 ; + } +if (func ==0x07 && idx ==1 && reg ==R_EAX ) { + ret &= ~CPUID_7_1_EAX_AMX_FP16 ; + } +if (func ==0x07 && idx ==1 && reg ==R_EDX ) { + ret &= ~CPUID_7_1_EDX_AMX_COMPLEX ; + }return ret; }
Signature · 1 line
2.34.1
Re: [PATCH 01/12] target/i386/mshv: disable AMX TILE features
Quoted 60 lines · Magnus Kulke
For the time being we disable AMX TILE in partition processor features and CPUID b/c AMX TILE XSAVE state (
XTILE_DATA) is 8KB, which exceeds the current fixed 4KB XSAVE buffer size.For now we filter it until buffer sizing is computed dynamically from CPUID.
Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 8 ++++++++ target/i386/mshv/mshv-cpu.c | 16 ++++++++++++++++ 2 files changed, 24 insertions(+)accel/mshv/mshv-all.c +8
-0@@ -203,6 +203,14 @@ static int create_partition(int mshv_fd, int *vm_fd)/* enable all */ disabled_xsave_features.as_uint64 =0 ; +/* +* AMX TILE XSAVE state (XTILE_DATA) is 8KB, which exceeds the +* current fixed 4KB XSAVE buffer size. +*/ + disabled_xsave_features.amx_tile_support =1 ; + disabled_xsave_features.amx_bf16_support =1 ; + disabled_xsave_features.amx_int8_support =1 ; + disabled_xsave_features.amx_fp16_support =1 ;/* * query host for supported processor features and disable unsupported target/i386/mshv/mshv-cpu.c +16
-0@@ -2156,6 +2156,22 @@ uint32_t mshv_get_supported_cpuid(uint32_t func, uint32_t idx, int reg)*/ ret &= ~CPUID_7_0_ECX_LA57 ; } +if (func ==0x07 && idx ==0 && reg ==R_EDX ) { +/* +* AMX TILE XSAVE state (XTILE_DATA) is 8KB, which exceeds the +* current fixed 4KB XSAVE buffer size. Filter until buffer +* sizing is computed dynamically from CPUID. +*/ + ret &= ~CPUID_7_0_EDX_AMX_TILE ; + ret &= ~CPUID_7_0_EDX_AMX_BF16 ; + ret &= ~CPUID_7_0_EDX_AMX_INT8 ; + } +if (func ==0x07 && idx ==1 && reg ==R_EAX ) { + ret &= ~CPUID_7_1_EAX_AMX_FP16 ; + } +if (func ==0x07 && idx ==1 && reg ==R_EDX ) { + ret &= ~CPUID_7_1_EDX_AMX_COMPLEX ; + }return ret; }Signature · 1 line
2.34.1
PATCH 2/12 accel/mshv: introduce SaveVMHandler
In reply to Magnus KulkeThis mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine. Currently it's just a skeleton.
--- accel/mshv/mshv-all.c | 7 +++++++ 1 file changed, 7 insertions(+)
accel/mshv/mshv-all.c +7 -0
@@ -39,6 +39,8 @@#include "system/mshv.h" #include "system/mshv_int.h" #include "system/reset.h" +#include "migration/qemu-file-types.h" +#include "migration/register.h" #include "trace.h" #include <err.h> #include <sys/ioctl.h>
@@ -544,6 +546,9 @@ static int mshv_init_vcpu(CPUState *cpu)return 0 ; } +static SaveVMHandlers savevm_mshv = { +}; +static int mshv_init (AccelState *as,MachineState *ms) {MshvState *s;
@@ -599,6 +604,8 @@ static int mshv_init(AccelState *as, MachineState *ms) 0, "mshv-memory");memory_listener_register (&mshv_io_listener, &address_space_io); +register_savevm_live ("mshv" ,0 ,1 , &savevm_mshv, s); +return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH 02/12] accel/mshv: introduce SaveVMHandler
Quoted 43 lines · Magnus Kulke
This mechanism is used to handle more imperative partition-wide steps that have to be taken as part of a migration routine. Currently it's just a skeleton.
Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 7 +++++++ 1 file changed, 7 insertions(+)accel/mshv/mshv-all.c +7
-0@@ -39,6 +39,8 @@#include "system/mshv.h" #include "system/mshv_int.h" #include "system/reset.h" +#include "migration/qemu-file-types.h" +#include "migration/register.h" #include "trace.h" #include <err.h> #include <sys/ioctl.h> @@ -544,6 +546,9 @@ static int mshv_init_vcpu(CPUState *cpu)return 0 ; } +static SaveVMHandlers savevm_mshv = { +}; +static int mshv_init (AccelState *as,MachineState *ms) {MshvState *s;@@ -599,6 +604,8 @@ static int mshv_init(AccelState *as, MachineState *ms) 0, "mshv-memory");memory_listener_register (&mshv_io_listener, &address_space_io); +register_savevm_live ("mshv" ,0 ,1 , &savevm_mshv, s); +return 0 ; }Signature · 1 line
2.34.1
PATCH 3/12 hw/i386/mshv: migrate REFERENCE_TIME
In reply to Magnus KulkeThis is a partition-wide state for which we use a dedicated hw clock facility, similar to KVM. We have to freeze the time for a partition before we are allowed to set it. We register a state change handler for the clock device and a post-load handler for migration state. In the post-load handler we toggle a flag that will set the reference time state on next state to "running" on the partition.
We can move the time freeze and reference-time ioctls/hvcalls to the clock module.
--- MAINTAINERS | 1 + accel/mshv/mshv-all.c | 67 ++------------ accel/mshv/trace-events | 1 + hw/i386/meson.build | 1 + hw/i386/mshv/clock.c | 189 +++++++++++++++++++++++++++++++++++++++ hw/i386/mshv/meson.build | 4 + include/system/mshv.h | 3 + 7 files changed, 208 insertions(+), 58 deletions(-) create mode 100644 hw/i386/mshv/clock.c create mode 100644 hw/i386/mshv/meson.build
MAINTAINERS +1 -0
@@ -613,6 +613,7 @@ R: Wei Liu <[email protected]> R: Doru Blânzeanu <[email protected]> S: Supported F: target/i386/mshv/ +F: hw/i386/mshv/ X86 Instruction Emulator M: Roman Bolshakov <[email protected]>
accel/mshv/mshv-all.c +9 -58
97 lines not on this page · raw message
accel/mshv/trace-events +1 -0
@@ -4,6 +4,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later mshv_start_vcpu_thread(const char* thread, uint32_t cpu) "thread=%s cpu_index=%d" +mshv_interrupt_exit_request(uint32_t cpu) "cpu_index=%d" mshv_set_memory(bool add, uint64_t gpa, uint64_t size, uint64_t user_addr, bool readonly, int ret) "add=%d gpa=0x%" PRIx64 " size=0x%" PRIx64 " user=0x%" PRIx64 " readonly=%d result=%d" mshv_mem_ioeventfd_add(uint64_t addr, uint32_t size, uint32_t data) "addr=0x%" PRIx64 " size=%d data=0x%x"
hw/i386/meson.build +1 -0
@@ -39,6 +39,7 @@ i386_ss.add(when: 'CONFIG_TDX', if_true: files('tdvf.c', 'tdvf-hob.c')) subdir('kvm') subdir('xen') +subdir('mshv') i386_ss.add_all(xenpv_ss)
hw/i386/mshv/clock.c new file +189 -0
189 lines not on this page · raw message
hw/i386/mshv/meson.build new file +4 -0
@@ -0,0 +1,4 @@ +i386_mshv_ss = ss.source_set() +i386_mshv_ss.add(files('clock.c')) + +i386_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
include/system/mshv.h +3 -0
@@ -55,6 +55,9 @@ DECLARE_INSTANCE_CHECKER(MshvState, MSHV_STATE,extern MshvState *mshv_state; +/* clock (partition reference time) */ +void mshv_clock_init (void ); +/* interrupt */ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint32_t vector, uint32_t vp_index, bool logical_destination_mode,
Signature · 1 line
2.34.1
Re: [PATCH 03/12] hw/i386/mshv: migrate REFERENCE_TIME
Quoted 386 lines · Magnus Kulke
This is a partition-wide state for which we use a dedicated hw clock facility, similar to KVM. We have to freeze the time for a partition before we are allowed to set it. We register a state change handler for the clock device and a post-load handler for migration state. In the post-load handler we toggle a flag that will set the reference time state on next state to "running" on the partition.
We can move the time freeze and reference-time ioctls/hvcalls to the clock module.
Signed-off-by Magnus Kulke <[email protected]>--- MAINTAINERS | 1 + accel/mshv/mshv-all.c | 67 ++------------ accel/mshv/trace-events | 1 + hw/i386/meson.build | 1 + hw/i386/mshv/clock.c | 189 +++++++++++++++++++++++++++++++++++++++ hw/i386/mshv/meson.build | 4 + include/system/mshv.h | 3 + 7 files changed, 208 insertions(+), 58 deletions(-) create mode 100644 hw/i386/mshv/clock.c create mode 100644 hw/i386/mshv/meson.buildMAINTAINERS +1
-0@@ -613,6 +613,7 @@ R: Wei Liu <[email protected]> R: Doru Blânzeanu <[email protected]> S: Supported F: target/i386/mshv/ +F: hw/i386/mshv/ X86 Instruction Emulator M: Roman Bolshakov <[email protected]>accel/mshv/mshv-all.c +9
-58@@ -60,54 +60,6 @@ static int init_mshv(int *mshv_fd)return 0 ; } -/* freeze 1 to pause, 0 to resume */ -static int set_time_freeze (int vm_fd,int freeze) -{ -int ret; -struct hv_input_set_partition_property in = {0 }; - in.property_code =HV_PARTITION_PROPERTY_TIME_FREEZE ; - in.property_value = freeze; - -struct mshv_root_hvcall args = {0 }; - args.code =HVCALL_SET_PARTITION_PROPERTY ; - args.in_sz =sizeof (in); - args.in_ptr = (uint64_t )∈ - - ret =mshv_hvcall (vm_fd, &args); -if (ret <0 ) { -error_report ("Failed to set time freeze" ); -return -1 ; - } - -return 0 ; -} - -static int pause_vm (int vm_fd) -{ -int ret; - - ret =set_time_freeze (vm_fd,1 ); -if (ret <0 ) { -error_report ("Failed to pause partition: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int resume_vm (int vm_fd) -{ -int ret; - - ret =set_time_freeze (vm_fd,0 ); -if (ret <0 ) { -error_report ("Failed to resume partition: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} -static int get_host_partition_property (int mshv_fd,uint32_t property_code,uint64_t *value) {@@ -330,9 +282,6 @@ static int create_vm(int mshv_fd, int *vm_fd)return -1 ; } -/* Always create a frozen partition */ -pause_vm (*vm_fd); -return 0 ; }@@ -578,13 +527,6 @@ static int mshv_init(AccelState *as, MachineState *ms)return -1 ; } - ret =resume_vm (vm_fd); -if (ret <0 ) { -close (mshv_fd); -close (vm_fd); -return -1 ; - } - s->vm = vm_fd; s->fd = mshv_fd;@@ -606,6 +548,8 @@ static int mshv_init(AccelState *as, MachineState *ms)register_savevm_live ("mshv" ,0 ,1 , &savevm_mshv, s); +mshv_clock_init (); +return 0 ; }@@ -643,6 +587,13 @@ static int mshv_cpu_exec(CPUState *cpu) cpu->vcpu_dirty = false; } +/* Corresponding store-release is in cpu_exit. */ +if (qatomic_load_acquire (&cpu->exit_request)) { +trace_mshv_interrupt_exit_request (cpu->cpu_index); + ret =EXCP_INTERRUPT ; +break ; + } + ret =mshv_run_vcpu (mshv_state->vm, cpu, &mshv_msg, &exit_reason);if (ret <0 ) {error_report ("Failed to run on vcpu %d" , cpu->cpu_index);accel/mshv/trace-events +1
-0@@ -4,6 +4,7 @@ # SPDX-License-Identifier: GPL-2.0-or-later mshv_start_vcpu_thread(const char* thread, uint32_t cpu) "thread=%s cpu_index=%d" +mshv_interrupt_exit_request(uint32_t cpu) "cpu_index=%d" mshv_set_memory(bool add, uint64_t gpa, uint64_t size, uint64_t user_addr, bool readonly, int ret) "add=%d gpa=0x%" PRIx64 " size=0x%" PRIx64 " user=0x%" PRIx64 " readonly=%d result=%d" mshv_mem_ioeventfd_add(uint64_t addr, uint32_t size, uint32_t data) "addr=0x%" PRIx64 " size=%d data=0x%x"hw/i386/meson.build +1
-0@@ -39,6 +39,7 @@ i386_ss.add(when: 'CONFIG_TDX', if_true: files('tdvf.c', 'tdvf-hob.c')) subdir('kvm') subdir('xen') +subdir('mshv') i386_ss.add_all(xenpv_ss)hw/i386/mshv/clock.c new file +189
-0@@ -0,0 +1,189 @@ +/* +* MSHV partition reference clock +* +* Copyright Microsoft, Corp. 2026 +* +* Authors: Magnus Kulke <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "qemu/error-report.h" +#include "migration/vmstate.h" +#include "system/runstate.h" +#include "hw/hyperv/hvhdk.h" +#include "hw/hyperv/hvhdk_mini.h" +#include "hw/hyperv/hvgdk.h" +#include "hw/hyperv/hvgdk_mini.h" +#include "linux/mshv.h" +#include "system/mshv.h" +#include "system/mshv_int.h" + +/* +* Partition reference clock (HV_PARTITION_PROPERTY_REFERENCE_TIME), captured +* when the VM is stopped and re-applied when it resumes. +* +* Mirrors hw/i386/kvm/clock.c. +*/ +typedef struct MshvClockState { +uint64_t ref_time; +bool ref_time_pending; +}MshvClockState ; + +static MshvClockState mshv_clock; + +static int mshv_get_reference_time (int vm_fd,uint64_t *ref_time) +{ +struct hv_input_get_partition_property in = {0 }; +struct hv_output_get_partition_property out = {0 }; +struct mshv_root_hvcall args = {0 }; +int ret; + + in.property_code =HV_PARTITION_PROPERTY_REFERENCE_TIME ; + + args.code =HVCALL_GET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + args.out_sz =sizeof (out); + args.out_ptr = (uint64_t )&out; + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to get reference time" ); +return -1 ; + } + + *ref_time = out.property_value; +return 0 ; +} + +static int mshv_set_reference_time (int vm_fd,uint64_t ref_time) +{ +struct hv_input_set_partition_property in = {0 }; +struct mshv_root_hvcall args = {0 }; +int ret; + + in.property_code =HV_PARTITION_PROPERTY_REFERENCE_TIME ; + in.property_value = ref_time; + + args.code =HVCALL_SET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to set reference time" ); +return -1 ; + } + +return 0 ; +} + +/* +* Freeze (freeze=1) or unfreeze (freeze=0) time for a partition. This will not +* pause/eject vCPU execution. It is assumed that the caller already has stopped +* the partition's vCPUs. +* +* NB: a partition's reference clock can only be written while the time is +* frozen. +*/ +static int mshv_set_time_freeze (int vm_fd,int freeze) +{ +struct hv_input_set_partition_property in = {0 }; +struct mshv_root_hvcall args = {0 }; +int ret; + + in.property_code =HV_PARTITION_PROPERTY_TIME_FREEZE ; + in.property_value = freeze; + + args.code =HVCALL_SET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + + ret =mshv_hvcall (vm_fd, &args); +if (ret <0 ) { +error_report ("Failed to set time freeze" ); +return -1 ; + } + +return 0 ; +} + +static void mshv_clock_vm_state_change (void *opaque,bool running, +RunState state) +{ +MshvClockState *s = opaque; +int vm_fd = mshv_state->vm; +int ret; + +if (running) { +/* Skip if we have nothing to restore, e.g. on initial boot. */ +if (!s->ref_time_pending) { +return ; + } + + ret =mshv_set_time_freeze (vm_fd,1 ); +if (ret <0 ) { +error_report ("Failed to freeze partition time on resume" ); +abort (); + } + +/* INVARIANT: reference time can only be written if time is frozen. */ + ret =mshv_set_reference_time (vm_fd, s->ref_time); +if (ret <0 ) { +error_report ("Failed to restore reference time on resume" ); +abort (); + } + +if (mshv_set_time_freeze (vm_fd,0 ) <0 ) { +error_report ("Failed to unfreeze partition time on resume" ); +abort (); + } + + s->ref_time_pending = false; + }else { +/* Skip if we already have a to-be-set ref time */ +if (s->ref_time_pending) { +return ; + } + + ret =mshv_get_reference_time (vm_fd, &s->ref_time); +if (ret <0 ) { +error_report ("Failed to capture reference time on stop" ); +abort (); + } + + s->ref_time_pending = true; + } +} + +/* +* The incoming reference time should be applied on the next resume before +* vCPUs start executing. +*/ +static int mshv_clock_post_load (void *opaque,int version_id) +{ +MshvClockState *s = opaque; + + s->ref_time_pending = true; + +return 0 ; +} + +static const VMStateDescription vmstate_mshv_clock = { + .name ="mshv-clock" , + .version_id =1 , + .minimum_version_id =1 , + .post_load = mshv_clock_post_load, + .fields = (const VMStateField []) { +VMSTATE_UINT64 (ref_time, MshvClockState), +VMSTATE_END_OF_LIST () + }, +}; + +void mshv_clock_init (void ) +{ +vmstate_register (NULL ,0 , &vmstate_mshv_clock, &mshv_clock); +qemu_add_vm_change_state_handler (mshv_clock_vm_state_change, &mshv_clock); +}hw/i386/mshv/meson.build new file +4
-0@@ -0,0 +1,4 @@ +i386_mshv_ss = ss.source_set() +i386_mshv_ss.add(files('clock.c')) + +i386_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)include/system/mshv.h +3
-0@@ -55,6 +55,9 @@ DECLARE_INSTANCE_CHECKER(MshvState, MSHV_STATE,extern MshvState *mshv_state; +/* clock (partition reference time) */ +void mshv_clock_init (void ); +/* interrupt */ int mshv_request_interrupt(MshvState *mshv_state, uint32_t interrupt_type, uint32_t vector, uint32_t vp_index, bool logical_destination_mode,Signature · 1 line
2.34.1
PATCH 4/12 accel/mshv: install dummy handler for SIG_IPI
In reply to Magnus KulkeThis is similar to HVF's implementation. We want to interrupt the blocking vcpu run. The self-kick was effectively a no-op for mshv.
--- accel/mshv/mshv-all.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
accel/mshv/mshv-all.c +6 -10
@@ -621,17 +621,13 @@ static int mshv_cpu_exec(CPUState *cpu) }/* -* The signal handler is triggered when QEMU's main thread receives a SIG_IPI -* (SIGUSR1). This signal causes the current CPU thread to be kicked, forcing a -* VM exit on the CPU. The VM exit generates an exit reason that breaks the loop -* (see mshv_cpu_exec). If the exit is due to a Ctrl+A+x command, the system -* will shut down. For other cases, the system will continue running. +* We need a dummy handler to make SIG_IPI a deliverable signal. The kernel +* handler will be woken up by the caught signal and instruct the hypervisor +* to suspend execution (the concrete mechanism differs between schedulers) +* and return to userspace. */ -static void sa_ipi_handler (int sig) +static void dummy_handler (int sig) { -/* TODO: call IOCTL to set_immediate_exit, once implemented. */ - -qemu_cpu_kick_self (); }static void init_signal (CPUState *cpu)
@@ -641,7 +637,7 @@ static void init_signal(CPUState *cpu)sigset_t set;memset (&sigact,0 ,sizeof (sigact)); - sigact.sa_handler = sa_ipi_handler; + sigact.sa_handler = dummy_handler;sigaction (SIG_IPI , &sigact,NULL );pthread_sigmask (SIG_BLOCK ,NULL , &set);
Signature · 1 line
2.34.1
Re: [PATCH 04/12] accel/mshv: install dummy handler for SIG_IPI
Quoted 46 lines · Magnus Kulke
This is similar to HVF's implementation. We want to interrupt the blocking vcpu run. The self-kick was effectively a no-op for mshv.
Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)accel/mshv/mshv-all.c +6
-10@@ -621,17 +621,13 @@ static int mshv_cpu_exec(CPUState *cpu) }/* -* The signal handler is triggered when QEMU's main thread receives a SIG_IPI -* (SIGUSR1). This signal causes the current CPU thread to be kicked, forcing a -* VM exit on the CPU. The VM exit generates an exit reason that breaks the loop -* (see mshv_cpu_exec). If the exit is due to a Ctrl+A+x command, the system -* will shut down. For other cases, the system will continue running. +* We need a dummy handler to make SIG_IPI a deliverable signal. The kernel +* handler will be woken up by the caught signal and instruct the hypervisor +* to suspend execution (the concrete mechanism differs between schedulers) +* and return to userspace. */ -static void sa_ipi_handler (int sig) +static void dummy_handler (int sig) { -/* TODO: call IOCTL to set_immediate_exit, once implemented. */ - -qemu_cpu_kick_self (); }static void init_signal (CPUState *cpu)@@ -641,7 +637,7 @@ static void init_signal(CPUState *cpu)sigset_t set;memset (&sigact,0 ,sizeof (sigact)); - sigact.sa_handler = sa_ipi_handler; + sigact.sa_handler = dummy_handler;sigaction (SIG_IPI , &sigact,NULL );pthread_sigmask (SIG_BLOCK ,NULL , &set);Signature · 1 line
2.34.1
PATCH 5/12 target/i386/mshv: migrate LAPIC state
In reply to Magnus KulkeThis change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM. MSHV currently keeps PIC/IOAPIC emulation in userspace, while LAPIC interrupt injection is handled through hypercalls.
We introduced dedicated apic infra in hw/i386/mshv to handle the migration and move lapic related functions from target/i386/mshv there. References have been the WHPX's whpx-apic implemenation and the mshv-ioctls crate's get_/set_lapic() impl for the mapping between MSHV/QEMU lapic state.
We are mapping the lapic state that we receive from the hypervisor to fields in APICCommonState. Common fields are used where feasible, with an mshv-specific MshvAPICState object that carries mshv-specific fields.
We have introduced a guard in pic_irq_request() that will early exit for the mshv accelerator, because mshv cannot take part in the userland path for legacy PIC interrupt injection.
The TSC_DEADLINE MSR is also migrated as part of LAPIC migration.
--- accel/mshv/mshv-all.c | 1 + hw/i386/meson.build | 2 +- hw/i386/mshv/apic.c | 395 +++++++++++++++++++++++++++++++++ hw/i386/mshv/meson.build | 1 + hw/i386/x86-cpu.c | 6 + include/hw/hyperv/hvgdk_mini.h | 2 + include/hw/i386/apic-msidef.h | 1 + include/system/mshv.h | 2 + include/system/mshv_int.h | 8 +- target/i386/cpu-apic.c | 3 + target/i386/mshv/mshv-cpu.c | 151 ++++--------- target/i386/mshv/msr.c | 2 + 12 files changed, 460 insertions(+), 114 deletions(-) create mode 100644 hw/i386/mshv/apic.c
accel/mshv/mshv-all.c +1 -0
@@ -813,6 +813,7 @@ static void mshv_accel_ops_class_init(ObjectClass *oc, const void *data) ops->synchronize_state = mshv_cpu_synchronize; ops->synchronize_pre_loadvm = mshv_cpu_synchronize_pre_loadvm; ops->cpus_are_resettable = mshv_cpus_are_resettable; + ops->cpu_thread_is_idle = mshv_vcpu_thread_is_idle; ops->handle_interrupt = generic_handle_interrupt; }
hw/i386/meson.build +1 -1
@@ -38,8 +38,8 @@ i386_ss.add(when: 'CONFIG_X86_FW_OVMF', if_true: files('pc_sysfw_ovmf.c'), i386_ss.add(when: 'CONFIG_TDX', if_true: files('tdvf.c', 'tdvf-hob.c')) subdir('kvm') -subdir('xen') subdir('mshv') +subdir('xen') i386_ss.add_all(xenpv_ss)
hw/i386/mshv/apic.c new file +395 -0
395 lines not on this page · raw message
hw/i386/mshv/meson.build +1 -0
@@ -1,4 +1,5 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files('clock.c')) +i386_mshv_ss.add(when: 'CONFIG_APIC', if_true: files('apic.c')) i386_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
hw/i386/x86-cpu.c +6 -0
@@ -22,6 +22,7 @@*/ #include "qemu/osdep.h" #include "system/whpx.h" +#include "system/mshv.h" #include "system/cpu-timers.h" #include "trace.h"
@@ -44,6 +45,11 @@ static void pic_irq_request(void *opaque, int irq, int level)X86CPU *cpu =X86_CPU (cs);trace_x86_pic_interrupt (irq, level); + +if (mshv_irqchip_in_kernel ()) { +return ; + } +if (cpu_is_apic_enabled (cpu->apic_state) && !kvm_irqchip_in_kernel () && !whpx_irqchip_in_kernel ()) {CPU_FOREACH (cs) {
include/hw/hyperv/hvgdk_mini.h +2 -0
@@ -168,6 +168,7 @@ typedef enum hv_register_name {/* Available */ HV_X64_REGISTER_SPEC_CTRL =0x00080084 , +HV_X64_REGISTER_TSC_DEADLINE =0x00080095 , HV_X64_REGISTER_TSC_ADJUST = 0x00080096,/* CET / Shadow Stack */
@@ -930,6 +931,7 @@ struct hv_cpuid {#define IA32_MSR_DEBUG_CTL 0x1D9#define IA32_MSR_SPEC_CTRL 0x00000048#define IA32_MSR_TSC_ADJUST 0x0000003b +#define IA32_MSR_TSC_DEADLINE 0x000006e0#define IA32_MSR_MISC_ENABLE 0x000001a0
include/hw/i386/apic-msidef.h +1 -0
@@ -13,6 +13,7 @@#define MSI_DATA_VECTOR_MASK 0x000000ff#define MSI_DATA_DELIVERY_MODE_SHIFT 8 +#define MSI_DATA_DELIVERY_MODE_MASK 7#define MSI_DATA_LEVEL_SHIFT 14#define MSI_DATA_TRIGGER_SHIFT 15
include/system/mshv.h +2 -0
@@ -41,9 +41,11 @@extern bool mshv_allowed;#define mshv_enabled () (mshv_allowed)#define mshv_msi_via_irqfd_enabled () mshv_enabled() +#define mshv_irqchip_in_kernel () mshv_enabled()#else /* CONFIG_MSHV_IS_POSSIBLE */ #define mshv_enabled () false#define mshv_msi_via_irqfd_enabled () mshv_enabled() +#define mshv_irqchip_in_kernel () mshv_enabled()#endif #define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv")
include/system/mshv_int.h +7 -1
@@ -105,10 +105,16 @@ void mshv_arch_amend_proc_features(void mshv_arch_disable_partition_proc_features (union hv_partition_processor_features *disabled_features);int mshv_arch_post_init_vm (int vm_fd); - +int mshv_get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state); +int mshv_set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state);typedef struct mshv_root_hvcall mshv_root_hvcall ;int mshv_hvcall (int fd,const mshv_root_hvcall *args); +/* apic */ +int mshv_init_lint (CPUState *cpu); +int mshv_set_lapic (const CPUState *cpu); +int mshv_get_lapic (CPUState *cpu); +/* memory */ typedef struct MshvMemoryRegion { uint64_t guest_phys_addr;
target/i386/cpu-apic.c +3 -0
@@ -14,6 +14,7 @@#include "system/hw_accel.h" #include "system/kvm.h" #include "system/xen.h" +#include "system/mshv.h" #include "system/address-spaces.h" #include "hw/core/qdev-properties.h" #include "hw/i386/apic_internal.h"
@@ -34,6 +35,8 @@ APICCommonClass *apic_get_class(Error **errp) apic_type ="xen-apic" ; }else if (whpx_irqchip_in_kernel ()) { apic_type ="whpx-apic" ; + }else if (mshv_enabled ()) { + apic_type ="mshv-apic" ; }return APIC_COMMON_CLASS (object_class_by_name (apic_type));
target/i386/mshv/mshv-cpu.c +39 -112
187 lines not on this page · raw message
target/i386/mshv/msr.c +2 -0
@@ -60,6 +60,8 @@ static const MshvMsrEnvMap msr_env_map[] = { offsetof(CPUX86State, tsc_aux) }, {IA32_MSR_TSC_ADJUST ,HV_X64_REGISTER_TSC_ADJUST , offsetof(CPUX86State, tsc_adjust) }, + { IA32_MSR_TSC_DEADLINE, HV_X64_REGISTER_TSC_DEADLINE, + offsetof(CPUX86State, tsc_deadline) },/* Hyper-V per-partition MSRs */ { HV_X64_MSR_HYPERCALL, HV_X64_REGISTER_HYPERCALL,
Signature · 1 line
2.34.1
Re: [PATCH 05/12] target/i386/mshv: migrate LAPIC state
Quoted 805 lines · Magnus Kulke
This change implements loading and storing the hyperv lapic state as part of the load/store routines for a vcpu.
The HyperV LAPIC is similar to the the split-irqchip in KVM. MSHV currently keeps PIC/IOAPIC emulation in userspace, while LAPIC interrupt injection is handled through hypercalls.
We introduced dedicated apic infra in hw/i386/mshv to handle the migration and move lapic related functions from target/i386/mshv there. References have been the WHPX's whpx-apic implemenation and the mshv-ioctls crate's get_/set_lapic() impl for the mapping between MSHV/QEMU lapic state.
We are mapping the lapic state that we receive from the hypervisor to fields in APICCommonState. Common fields are used where feasible, with an mshv-specific MshvAPICState object that carries mshv-specific fields.
We have introduced a guard in
pic_irq_request()that will early exit for the mshv accelerator, because mshv cannot take part in the userland path for legacy PIC interrupt injection.The
TSC_DEADLINEMSR is also migrated as part of LAPIC migration.Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 1 + hw/i386/meson.build | 2 +- hw/i386/mshv/apic.c | 395 +++++++++++++++++++++++++++++++++ hw/i386/mshv/meson.build | 1 + hw/i386/x86-cpu.c | 6 + include/hw/hyperv/hvgdk_mini.h | 2 + include/hw/i386/apic-msidef.h | 1 + include/system/mshv.h | 2 + include/system/mshv_int.h | 8 +- target/i386/cpu-apic.c | 3 + target/i386/mshv/mshv-cpu.c | 151 ++++--------- target/i386/mshv/msr.c | 2 + 12 files changed, 460 insertions(+), 114 deletions(-) create mode 100644 hw/i386/mshv/apic.caccel/mshv/mshv-all.c +1
-0@@ -813,6 +813,7 @@ static void mshv_accel_ops_class_init(ObjectClass *oc, const void *data) ops->synchronize_state = mshv_cpu_synchronize; ops->synchronize_pre_loadvm = mshv_cpu_synchronize_pre_loadvm; ops->cpus_are_resettable = mshv_cpus_are_resettable; + ops->cpu_thread_is_idle = mshv_vcpu_thread_is_idle; ops->handle_interrupt = generic_handle_interrupt; }hw/i386/meson.build +1
-1@@ -38,8 +38,8 @@ i386_ss.add(when: 'CONFIG_X86_FW_OVMF', if_true: files('pc_sysfw_ovmf.c'), i386_ss.add(when: 'CONFIG_TDX', if_true: files('tdvf.c', 'tdvf-hob.c')) subdir('kvm') -subdir('xen') subdir('mshv') +subdir('xen') i386_ss.add_all(xenpv_ss)hw/i386/mshv/apic.c new file +395
-0@@ -0,0 +1,395 @@ +/* +* MSHV in-kernel APIC support +* +* Copyright Microsoft, Corp. 2026 +* +* Authors: Magnus Kulke <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "qemu/module.h" +#include "qemu/memalign.h" +#include "qemu/error-report.h" +#include "hw/i386/apic_internal.h" +#include "hw/i386/apic-msidef.h" +#include "hw/pci/msi.h" +#include "migration/vmstate.h" +#include "qemu/typedefs.h" +#include "system/hw_accel.h" +#include "system/mshv.h" +#include "system/mshv_int.h" + +typedef struct hv_local_interrupt_controller_state +hv_local_interrupt_controller_state ; + +#define TYPE_MSHV_APIC "mshv-apic" +OBJECT_DECLARE_SIMPLE_TYPE(MshvAPICState, MSHV_APIC) + +struct MshvAPICState { +APICCommonState parent_obj; + +uint32_t apic_version; +uint32_t apic_lvt_cmci; +uint32_t apic_error_status; +uint32_t apic_counter_value; +uint32_t apic_remote_read; +}; + +static int get_lapic (int cpu_fd, +struct hv_local_interrupt_controller_state *state) +{ +int ret; +size_t size =4096 ; +/* buffer aligned to 4k, as *state requires that */ +void *buffer =qemu_memalign (size, size); +struct mshv_get_set_vp_state mshv_state = {0 }; + + mshv_state.buf_ptr = (uint64_t ) buffer; + mshv_state.buf_sz = size; + mshv_state.type =MSHV_VP_STATE_LAPIC ; + + ret =mshv_get_vp_state (cpu_fd, &mshv_state); +if (ret ==0 ) { +memcpy (state, buffer,sizeof (*state)); + } +qemu_vfree (buffer); +if (ret <0 ) { +error_report ("failed to get lapic" ); +return -1 ; + } + +return 0 ; +} + +static int set_lapic (int cpu_fd, +const struct hv_local_interrupt_controller_state *state) +{ +int ret; +size_t size =4096 ; +/* buffer aligned to 4k, as *state requires that */ +void *buffer =qemu_memalign (size, size); +struct mshv_get_set_vp_state mshv_state = {0 }; + +if (!state) { +error_report ("lapic state is NULL" ); +return -1 ; + } +memcpy (buffer, state,sizeof (*state)); + + mshv_state.buf_ptr = (uint64_t ) buffer; + mshv_state.buf_sz = size; + mshv_state.type =MSHV_VP_STATE_LAPIC ; + + ret =mshv_set_vp_state (cpu_fd, &mshv_state); +qemu_vfree (buffer); +if (ret <0 ) { +error_report ("failed to set lapic: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static void populate_apic_state (CPUState *cpu, +const hv_local_interrupt_controller_state *hv) +{ +X86CPU *x86cpu =X86_CPU (cpu); +MshvAPICState *ms =MSHV_APIC (x86cpu->apic_state); +APICCommonState *s = &ms->parent_obj; +size_t i; + +/* +* x2APIC: +* - APIC ID is the full 32-bit initial_apic_id +* - LDR is read-only, architecturally derived from the ID +* - DFR does not exist in x2APIC mode +*/ +if (is_x2apic_mode (s)) { + s->initial_apic_id = hv->apic_id; + }else { + s->id = hv->apic_id >>24 ; + s->log_dest = hv->apic_ldr >>24 ; + s->dest_mode = hv->apic_dfr >>28 ; + } + ms->apic_version = hv->apic_version; + s->spurious_vec = hv->apic_spurious; +for (i =0 ; i <8 ; i++) { + s->isr[i] = hv->apic_isr[i]; + s->tmr[i] = hv->apic_tmr[i]; + s->irr[i] = hv->apic_irr[i]; + } + s->esr = hv->apic_esr; + s->icr[1 ] = hv->apic_icr_high; + s->icr[0 ] = hv->apic_icr_low; + + s->lvt[APIC_LVT_TIMER ] = hv->apic_lvt_timer; + s->lvt[APIC_LVT_THERMAL ] = hv->apic_lvt_thermal; + s->lvt[APIC_LVT_PERFORM ] = hv->apic_lvt_perfmon; + s->lvt[APIC_LVT_LINT0 ] = hv->apic_lvt_lint0; + s->lvt[APIC_LVT_LINT1 ] = hv->apic_lvt_lint1; + s->lvt[APIC_LVT_ERROR ] = hv->apic_lvt_error; + ms->apic_lvt_cmci = hv->apic_lvt_cmci; + + ms->apic_error_status = hv->apic_error_status; + s->initial_count = hv->apic_initial_count; + ms->apic_counter_value = hv->apic_counter_value; + s->divide_conf = hv->apic_divide_configuration; + ms->apic_remote_read = hv->apic_remote_read; +} + +static uint32_t set_apic_delivery_mode (uint32_t reg,uint32_t mode) +{ +return ((reg) & ~0x700 ) | ((mode) <<8 ); +} + +int mshv_init_lint (CPUState *cpu) +{ +uint32_t *lvt_lint0, *lvt_lint1; +int cpu_fd =mshv_vcpufd (cpu); +int ret; +struct hv_local_interrupt_controller_state lapic_state = {0 }; + + ret =get_lapic (cpu_fd, &lapic_state); +if (ret <0 ) { +return ret; + } + + lvt_lint0 = &lapic_state.apic_lvt_lint0; + *lvt_lint0 =set_apic_delivery_mode (*lvt_lint0,APIC_DM_EXTINT ); + + lvt_lint1 = &lapic_state.apic_lvt_lint1; + *lvt_lint1 =set_apic_delivery_mode (*lvt_lint1,APIC_DM_NMI ); + +/* TODO: should we skip setting lapic if the values are the same? */ + + ret =set_lapic (cpu_fd, &lapic_state); +if (ret <0 ) { +return -1 ; + } + +populate_apic_state (cpu, &lapic_state); + +return 0 ; +} + +static void populate_hv_lapic_state (hv_local_interrupt_controller_state *hv, +const CPUState *cpu) +{ +uint32_t x2apic_id; +X86CPU *x86cpu =X86_CPU (cpu); +MshvAPICState *ms =MSHV_APIC (x86cpu->apic_state); +APICCommonState *s = &ms->parent_obj; +size_t i; + +/* +* x2APIC: +* - APIC ID is the full 32-bit initial_apic_id +* - LDR is read-only, architecturally derived from the ID +* - DFR does not exist in x2APIC mode +*/ +if (is_x2apic_mode (s)) { + x2apic_id = s->initial_apic_id; + + hv->apic_id = x2apic_id; + hv->apic_ldr = ((x2apic_id >>4 ) <<16 ) | (1 << (x2apic_id &0xf )); + hv->apic_dfr =0 ; + }else { + hv->apic_id = s->id <<24 ; + hv->apic_ldr = s->log_dest <<24 ; + hv->apic_dfr = s->dest_mode <<28 |0x0fffffff ; + } + hv->apic_version = ms->apic_version; + hv->apic_spurious = s->spurious_vec; +for (i =0 ; i <8 ; i++) { + hv->apic_isr[i] = s->isr[i]; + hv->apic_tmr[i] = s->tmr[i]; + hv->apic_irr[i] = s->irr[i]; + } + hv->apic_esr = s->esr; + hv->apic_icr_high = s->icr[1 ]; + hv->apic_icr_low = s->icr[0 ]; + + hv->apic_lvt_timer = s->lvt[APIC_LVT_TIMER ]; + hv->apic_lvt_thermal = s->lvt[APIC_LVT_THERMAL ]; + hv->apic_lvt_perfmon = s->lvt[APIC_LVT_PERFORM ]; + hv->apic_lvt_lint0 = s->lvt[APIC_LVT_LINT0 ]; + hv->apic_lvt_lint1 = s->lvt[APIC_LVT_LINT1 ]; + hv->apic_lvt_error = s->lvt[APIC_LVT_ERROR ]; + hv->apic_lvt_cmci = ms->apic_lvt_cmci; + + hv->apic_error_status = ms->apic_error_status; + hv->apic_initial_count = s->initial_count; + hv->apic_counter_value = ms->apic_counter_value; + hv->apic_divide_configuration = s->divide_conf; + hv->apic_remote_read = ms->apic_remote_read; +} + +int mshv_set_lapic (const CPUState *cpu) +{ +int cpu_fd =mshv_vcpufd (cpu); +struct hv_local_interrupt_controller_state lapic_state = {0 }; + +populate_hv_lapic_state (&lapic_state, cpu); + +return set_lapic (cpu_fd, &lapic_state); +} + +int mshv_get_lapic (CPUState *cpu) +{ +int cpu_fd =mshv_vcpufd (cpu); +int ret; +struct hv_local_interrupt_controller_state lapic_state = {0 }; + + ret =get_lapic (cpu_fd, &lapic_state); +if (ret <0 ) { +return -1 ; + } + +populate_apic_state (cpu, &lapic_state); + +return 0 ; +} + +static int mshv_apic_set_base (APICCommonState *s,uint64_t val) +{ + s->apicbase = val; + +return 0 ; +} + +static void mshv_apic_set_tpr (APICCommonState *s,uint8_t val) +{ + s->tpr = (val &APIC_PR_SUB_CLASS ) <<APIC_PR_CLASS_SHIFT ; +} + +static uint8_t mshv_apic_get_tpr (APICCommonState *s) +{ +return s->tpr >>APIC_PR_CLASS_SHIFT ; +} + +static void mshv_apic_external_nmi (APICCommonState *s) +{ +} + +static void mshv_apic_vapic_base_update (APICCommonState *s) +{ +} + +static void mshv_send_msi (MSIMessage *msi) +{ +uint64_t addr; +uint32_t data, dest; +uint8_t vector, dest_mode, trigger_mode, delivery; + + addr = msi->address; + data = msi->data; + dest = (addr &MSI_ADDR_DEST_ID_MASK ) >>MSI_ADDR_DEST_ID_SHIFT | + (addr >>32 ); + vector = (data &MSI_DATA_VECTOR_MASK ) >>MSI_DATA_VECTOR_SHIFT ; + dest_mode = (addr >>MSI_ADDR_DEST_MODE_SHIFT ) &0x1 ; + trigger_mode = (data >>MSI_DATA_TRIGGER_SHIFT ) &0x1 ; + delivery = (data >>MSI_DATA_DELIVERY_MODE_SHIFT ) & +MSI_DATA_DELIVERY_MODE_MASK ; + +mshv_request_interrupt (mshv_state, delivery, vector, dest, dest_mode, + trigger_mode); +} + +static uint64_t mshv_apic_mem_read (void *opaque,hwaddr addr, +unsigned size) +{ +return UINT64_MAX ; +} + +static void mshv_apic_mem_write (void *opaque,hwaddr addr, +uint64_t data,unsigned size) +{ +MSIMessage msg = { .address = addr, .data = data }; + +mshv_send_msi (&msg); +} + +static const MemoryRegionOps mshv_apic_io_ops = { + .read = mshv_apic_mem_read, + .write = mshv_apic_mem_write, + .endianness =DEVICE_LITTLE_ENDIAN , +}; + +static void mshv_apic_reset (APICCommonState *s) +{ + s->wait_for_sipi =0 ; +} + +static const VMStateDescription vmstate_mshv_apic = { + .name ="mshv-apic" , + .version_id =1 , + .minimum_version_id =1 , + .fields = (const VMStateField []) { +VMSTATE_UINT32 (apic_version, MshvAPICState), +VMSTATE_UINT32 (apic_lvt_cmci, MshvAPICState), +VMSTATE_UINT32 (apic_error_status, MshvAPICState), +VMSTATE_UINT32 (apic_counter_value, MshvAPICState), +VMSTATE_UINT32 (apic_remote_read, MshvAPICState), +VMSTATE_END_OF_LIST () + } +}; + +static void mshv_apic_realize (DeviceState *dev,Error **errp) +{ +APICCommonState *s =APIC_COMMON (dev); +MshvAPICState *ms =MSHV_APIC (dev); + +memory_region_init_io (&s->io_memory,OBJECT (s), &mshv_apic_io_ops, s, +"mshv-apic-msi" ,APIC_SPACE_SIZE ); + + msi_nonbroken = true; + +/* +* We register this state explicity, rather than going via dc->vmsd. +* The auto-wiring would register the state with +* instance_id == VMSTATE_INSTANCE_ID_ANY, which for the APIC doesn't +* work, b/c the ID carries semantic meaning for restoring the state +* on the destination (which vcpu it belongs to). +*/ +vmstate_register_with_alias_id (NULL , + s->initial_apic_id, &vmstate_mshv_apic, ms, +-1 ,0 ,NULL ); +} + +static void mshv_apic_unrealize (DeviceState *dev) +{ +MshvAPICState *ms =MSHV_APIC (dev); + +vmstate_unregister (NULL , &vmstate_mshv_apic, ms); +} + +static void mshv_apic_class_init (ObjectClass *klass,const void *data) +{ +APICCommonClass *k =APIC_COMMON_CLASS (klass); + + k->realize = mshv_apic_realize; + k->unrealize = mshv_apic_unrealize; + k->reset = mshv_apic_reset; + k->set_base = mshv_apic_set_base; + k->set_tpr = mshv_apic_set_tpr; + k->get_tpr = mshv_apic_get_tpr; + k->external_nmi = mshv_apic_external_nmi; + k->vapic_base_update = mshv_apic_vapic_base_update; + k->send_msi = mshv_send_msi; +} + +static const TypeInfo mshv_apic_info = { + .name =TYPE_MSHV_APIC , + .parent =TYPE_APIC_COMMON , + .instance_size =sizeof (MshvAPICState), + .class_init = mshv_apic_class_init, +}; + +static void mshv_apic_register_types (void ) +{ +type_register_static (&mshv_apic_info); +} + +type_init (mshv_apic_register_types)hw/i386/mshv/meson.build +1
-0@@ -1,4 +1,5 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files('clock.c')) +i386_mshv_ss.add(when: 'CONFIG_APIC', if_true: files('apic.c')) i386_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)hw/i386/x86-cpu.c +6
-0@@ -22,6 +22,7 @@*/ #include "qemu/osdep.h" #include "system/whpx.h" +#include "system/mshv.h" #include "system/cpu-timers.h" #include "trace.h" @@ -44,6 +45,11 @@ static void pic_irq_request(void *opaque, int irq, int level)X86CPU *cpu =X86_CPU (cs);trace_x86_pic_interrupt (irq, level); + +if (mshv_irqchip_in_kernel ()) { +return ; + } +if (cpu_is_apic_enabled (cpu->apic_state) && !kvm_irqchip_in_kernel () && !whpx_irqchip_in_kernel ()) {CPU_FOREACH (cs) {include/hw/hyperv/hvgdk_mini.h +2
-0@@ -168,6 +168,7 @@ typedef enum hv_register_name {/* Available */ HV_X64_REGISTER_SPEC_CTRL =0x00080084 , +HV_X64_REGISTER_TSC_DEADLINE =0x00080095 , HV_X64_REGISTER_TSC_ADJUST = 0x00080096,/* CET / Shadow Stack */ @@ -930,6 +931,7 @@ struct hv_cpuid {#define IA32_MSR_DEBUG_CTL 0x1D9#define IA32_MSR_SPEC_CTRL 0x00000048#define IA32_MSR_TSC_ADJUST 0x0000003b +#define IA32_MSR_TSC_DEADLINE 0x000006e0#define IA32_MSR_MISC_ENABLE 0x000001a0include/hw/i386/apic-msidef.h +1
-0@@ -13,6 +13,7 @@#define MSI_DATA_VECTOR_MASK 0x000000ff#define MSI_DATA_DELIVERY_MODE_SHIFT 8 +#define MSI_DATA_DELIVERY_MODE_MASK 7#define MSI_DATA_LEVEL_SHIFT 14#define MSI_DATA_TRIGGER_SHIFT 15include/system/mshv.h +2
-0@@ -41,9 +41,11 @@extern bool mshv_allowed;#define mshv_enabled () (mshv_allowed)#define mshv_msi_via_irqfd_enabled () mshv_enabled() +#define mshv_irqchip_in_kernel () mshv_enabled()#else /* CONFIG_MSHV_IS_POSSIBLE */ #define mshv_enabled () false#define mshv_msi_via_irqfd_enabled () mshv_enabled() +#define mshv_irqchip_in_kernel () mshv_enabled()#endif #define TYPE_MSHV_ACCEL ACCEL_CLASS_NAME("mshv")include/system/mshv_int.h +7
-1@@ -105,10 +105,16 @@ void mshv_arch_amend_proc_features(void mshv_arch_disable_partition_proc_features (union hv_partition_processor_features *disabled_features);int mshv_arch_post_init_vm (int vm_fd); - +int mshv_get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state); +int mshv_set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state);typedef struct mshv_root_hvcall mshv_root_hvcall ;int mshv_hvcall (int fd,const mshv_root_hvcall *args); +/* apic */ +int mshv_init_lint (CPUState *cpu); +int mshv_set_lapic (const CPUState *cpu); +int mshv_get_lapic (CPUState *cpu); +/* memory */ typedef struct MshvMemoryRegion { uint64_t guest_phys_addr;target/i386/cpu-apic.c +3
-0@@ -14,6 +14,7 @@#include "system/hw_accel.h" #include "system/kvm.h" #include "system/xen.h" +#include "system/mshv.h" #include "system/address-spaces.h" #include "hw/core/qdev-properties.h" #include "hw/i386/apic_internal.h" @@ -34,6 +35,8 @@ APICCommonClass *apic_get_class(Error **errp) apic_type ="xen-apic" ; }else if (whpx_irqchip_in_kernel ()) { apic_type ="whpx-apic" ; + }else if (mshv_enabled ()) { + apic_type ="mshv-apic" ; }return APIC_COMMON_CLASS (object_class_by_name (apic_type));target/i386/mshv/mshv-cpu.c +39
-112@@ -21,7 +21,6 @@#include "hw/hyperv/hvgdk.h" #include "hw/hyperv/hvgdk_mini.h" #include "hw/hyperv/hvhdk_mini.h" -#include "hw/i386/apic_internal.h" #include "cpu.h" #include "host-cpu.h" @@ -955,6 +954,11 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } + ret =mshv_get_lapic (cpu); +if (ret <0 ) { +return ret; + } + ret =mshv_get_msrs (cpu);if (ret <0 ) {return ret;@@ -1377,116 +1381,6 @@ static int set_xc_reg(const CPUState *cpu)return 0 ; } -static int get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state) -{ -int ret; - - ret =ioctl (cpu_fd,MSHV_GET_VP_STATE , state); -if (ret <0 ) { -error_report ("failed to get partition state: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int get_lapic (const CPUState *cpu, -struct hv_local_interrupt_controller_state *state) -{ -int ret; -size_t size =4096 ; -/* buffer aligned to 4k, as *state requires that */ -void *buffer =qemu_memalign (size, size); -struct mshv_get_set_vp_state mshv_state = {0 }; -int cpu_fd =mshv_vcpufd (cpu); - - mshv_state.buf_ptr = (uint64_t ) buffer; - mshv_state.buf_sz = size; - mshv_state.type =MSHV_VP_STATE_LAPIC ; - - ret =get_vp_state (cpu_fd, &mshv_state); -if (ret ==0 ) { -memcpy (state, buffer,sizeof (*state)); - } -qemu_vfree (buffer); -if (ret <0 ) { -error_report ("failed to get lapic" ); -return -1 ; - } - -return 0 ; -} - -static uint32_t set_apic_delivery_mode (uint32_t reg,uint32_t mode) -{ -return ((reg) & ~0x700 ) | ((mode) <<8 ); -} - -static int set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state) -{ -int ret; - - ret =ioctl (cpu_fd,MSHV_SET_VP_STATE , state); -if (ret <0 ) { -error_report ("failed to set partition state: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int set_lapic (const CPUState *cpu, -const struct hv_local_interrupt_controller_state *state) -{ -int ret; -size_t size =4096 ; -/* buffer aligned to 4k, as *state requires that */ -void *buffer =qemu_memalign (size, size); -struct mshv_get_set_vp_state mshv_state = {0 }; -int cpu_fd =mshv_vcpufd (cpu); - -if (!state) { -error_report ("lapic state is NULL" ); -return -1 ; - } -memcpy (buffer, state,sizeof (*state)); - - mshv_state.buf_ptr = (uint64_t ) buffer; - mshv_state.buf_sz = size; - mshv_state.type =MSHV_VP_STATE_LAPIC ; - - ret =set_vp_state (cpu_fd, &mshv_state); -qemu_vfree (buffer); -if (ret <0 ) { -error_report ("failed to set lapic: %s" ,strerror (errno)); -return -1 ; - } - -return 0 ; -} - -static int init_lint (const CPUState *cpu) -{ -int ret; -uint32_t *lvt_lint0, *lvt_lint1; - -struct hv_local_interrupt_controller_state lapic_state = {0 }; - ret =get_lapic (cpu, &lapic_state); -if (ret <0 ) { -return ret; - } - - lvt_lint0 = &lapic_state.apic_lvt_lint0; - *lvt_lint0 =set_apic_delivery_mode (*lvt_lint0,APIC_DM_EXTINT ); - - lvt_lint1 = &lapic_state.apic_lvt_lint1; - *lvt_lint1 =set_apic_delivery_mode (*lvt_lint1,APIC_DM_NMI ); - -/* TODO: should we skip setting lapic if the values are the same? */ - -return set_lapic (cpu, &lapic_state); -} -int mshv_arch_store_vcpu_state (const CPUState *cpu) {int ret;@@ -1511,6 +1405,12 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } +/* INVARIANT: special regs (APIC_BASE) must be restored before LAPIC */ + ret =mshv_set_lapic (cpu); +if (ret <0 ) { +return ret; + } + ret =mshv_set_msrs (cpu);if (ret <0 ) {return ret;@@ -2099,7 +1999,7 @@ void mshv_arch_init_vcpu(CPUState *cpu) ret =mshv_init_msrs (cpu);assert (ret ==0 ); - ret =init_lint (cpu); + ret =mshv_init_lint (cpu);assert (ret ==0 ); }@@ -2246,6 +2146,33 @@ static void mshv_cpu_xsave_init(void) } } +int mshv_set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state) +{ +int ret; + + ret =ioctl (cpu_fd,MSHV_SET_VP_STATE , state); +if (ret <0 ) { +error_report ("failed to set partition state: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + + +int mshv_get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state) +{ +int ret; + + ret =ioctl (cpu_fd,MSHV_GET_VP_STATE , state); +if (ret <0 ) { +error_report ("failed to get partition state: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} +static void mshv_cpu_instance_init (CPUState *cs) {X86CPU *cpu =X86_CPU (cs);target/i386/mshv/msr.c +2
-0@@ -60,6 +60,8 @@ static const MshvMsrEnvMap msr_env_map[] = { offsetof(CPUX86State, tsc_aux) }, {IA32_MSR_TSC_ADJUST ,HV_X64_REGISTER_TSC_ADJUST , offsetof(CPUX86State, tsc_adjust) }, + { IA32_MSR_TSC_DEADLINE, HV_X64_REGISTER_TSC_DEADLINE, + offsetof(CPUX86State, tsc_deadline) },/* Hyper-V per-partition MSRs */ { HV_X64_MSR_HYPERCALL, HV_X64_REGISTER_HYPERCALL,Signature · 1 line
2.34.1
PATCH 6/12 target/i386/mshv: migrate Synic SINT MSRs
In reply to Magnus KulkeMigrate HyperV SynIC SINT MSRs. We can only read/write those if SCONTROL is enabled in the guest, hence we have to split the SINT MSR out and make reading/writing them dependent on that MSR.
--- target/i386/mshv/msr.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
target/i386/mshv/msr.c +41 -0
@@ -331,6 +331,8 @@ int mshv_get_msrs(CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];size_t i, j;uint32_t name; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);
@@ -357,6 +359,27 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs); +/* Read SINT MSRs only if SynIC is enabled */ + synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + } + + ret =mshv_get_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to get SynIC SINT MSRs" ); +return -errno; + } + +for (i =0 ; i <HV_SINT_COUNT ; i++) { +uint64_t hv_sint_value = assocs[i].value.reg64; + x86cpu->env.msr_hv_synic_sint[i] = hv_sint_value; + } + } +return 0 ; }
@@ -391,6 +414,8 @@ int mshv_set_msrs(const CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];int ret;size_t i, j; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);
@@ -423,5 +448,21 @@ int mshv_set_msrs(const CPUState *cpu)return -errno; } +/* SINT MSRs can only be written if SCONTROL has been set, so we split */ +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + assocs[i].value.reg64 = x86cpu->env.msr_hv_synic_sint[i]; + } + + ret =mshv_set_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to set SynIC SINT MSRs" ); +return -errno; + } + } +return 0 ; }
Signature · 1 line
2.34.1
Re: [PATCH 06/12] target/i386/mshv: migrate Synic SINT MSRs
Quoted 83 lines · Magnus Kulke
Migrate HyperV SynIC SINT MSRs. We can only read/write those if SCONTROL is enabled in the guest, hence we have to split the SINT MSR out and make reading/writing them dependent on that MSR.
Signed-off-by Magnus Kulke <[email protected]>--- target/i386/mshv/msr.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)target/i386/mshv/msr.c +41
-0@@ -331,6 +331,8 @@ int mshv_get_msrs(CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];size_t i, j;uint32_t name; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);@@ -357,6 +359,27 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs); +/* Read SINT MSRs only if SynIC is enabled */ + synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + } + + ret =mshv_get_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to get SynIC SINT MSRs" ); +return -errno; + } + +for (i =0 ; i <HV_SINT_COUNT ; i++) { +uint64_t hv_sint_value = assocs[i].value.reg64; + x86cpu->env.msr_hv_synic_sint[i] = hv_sint_value; + } + } +return 0 ; }@@ -391,6 +414,8 @@ int mshv_set_msrs(const CPUState *cpu)struct hv_register_assoc assocs[MSHV_MSR_TOTAL_COUNT ];int ret;size_t i, j; +X86CPU *x86cpu =X86_CPU (cpu); +bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);@@ -423,5 +448,21 @@ int mshv_set_msrs(const CPUState *cpu)return -errno; } +/* SINT MSRs can only be written if SCONTROL has been set, so we split */ +if (synic_enabled) { +QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT ); + +for (i =0 ; i <HV_SINT_COUNT ; i++) { + assocs[i].name =HV_REGISTER_SINT0 + i; + assocs[i].value.reg64 = x86cpu->env.msr_hv_synic_sint[i]; + } + + ret =mshv_set_generic_regs (cpu, assocs,HV_SINT_COUNT ); +if (ret <0 ) { +error_report ("Failed to set SynIC SINT MSRs" ); +return -errno; + } + } +return 0 ; }Signature · 1 line
2.34.1
PATCH 7/12 target/i386/mshv: migrate SIMP and SIEFP state
In reply to Magnus KulkeThis part SynIC state is retrieved from the hypervisor via aligned state pages:
- Add new synic source file
- Centralize the synic_enabled() check
- r/w pages from the hyper via aligned pages
- only handle pages when synic is enabled
- add buffers for migration to VM state
--- include/system/mshv_int.h | 7 ++ target/i386/cpu.h | 5 ++ target/i386/machine.c | 26 ++++++ target/i386/mshv/meson.build | 1 + target/i386/mshv/mshv-cpu.c | 64 +++++++++++++++ target/i386/mshv/msr.c | 7 +- target/i386/mshv/synic.c | 155 +++++++++++++++++++++++++++++++++++ 7 files changed, 260 insertions(+), 5 deletions(-) create mode 100644 target/i386/mshv/synic.c
include/system/mshv_int.h +7 -0
@@ -138,4 +138,11 @@ int mshv_init_msrs(const CPUState *cpu);int mshv_get_msrs (CPUState *cpu);int mshv_set_msrs (const CPUState *cpu); +/* synic */ +int mshv_get_simp (int cpu_fd,uint8_t *page); +int mshv_set_simp (int cpu_fd,const uint8_t *page); +int mshv_get_siefp (int cpu_fd,uint8_t *page); +int mshv_set_siefp (int cpu_fd,const uint8_t *page); +bool mshv_synic_enabled (const CPUState *cpu); +#endif
target/i386/cpu.h +5 -0
@@ -33,6 +33,7 @@#include "qemu/cpu-float.h" #include "qemu/timer.h" #include "standard-headers/asm-x86/kvm_para.h" +#include "hw/hyperv/hvgdk_mini.h" #define XEN_NR_VIRQS 24
@@ -2299,6 +2300,10 @@ typedef struct CPUArchState {#if defined(CONFIG_HVF ) || defined(CONFIG_MSHV ) || defined(CONFIG_WHPX )void *emu_mmio_buf;#endif +#if defined(CONFIG_MSHV ) +uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +#endif uint64_t mcg_cap;uint64_t mcg_ctl;
target/i386/machine.c +26 -0
@@ -951,6 +951,29 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { } }; +#ifdef CONFIG_MSHV +static bool mshv_synic_vp_state_needed (void *opaque) +{ +X86CPU *cpu = opaque; +CPUX86State *env = &cpu->env; + +/* Only migrate SIMP/SIEFP if SynIC is enabled */ +return env->msr_hv_synic_control &1 ; +} + +static const VMStateDescription vmstate_mshv_synic_vp_state = { + .name ="cpu/mshv_synic_vp_state" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synic_vp_state_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_simp_page,X86CPU ), +VMSTATE_BUFFER (env.hv_siefp_page,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +#endif +static bool avx512_needed (void *opaque) {X86CPU *cpu = opaque;
@@ -1915,6 +1938,9 @@ const VMStateDescription vmstate_x86_cpu = { &vmstate_cet,#ifdef TARGET_X86_64 &vmstate_apx, +#endif +#ifdef CONFIG_MSHV + &vmstate_mshv_synic_vp_state, #endifNULL }
target/i386/mshv/meson.build +1 -0
@@ -3,6 +3,7 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files( 'mshv-cpu.c', 'msr.c', + 'synic.c', )) i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)
target/i386/mshv/mshv-cpu.c +64 -0
88 lines not on this page · raw message
target/i386/mshv/msr.c +2 -5
@@ -332,7 +332,6 @@ int mshv_get_msrs(CPUState *cpu)size_t i, j;uint32_t name;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);
@@ -360,8 +359,7 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs);/* Read SINT MSRs only if SynIC is enabled */ - synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {
@@ -415,7 +413,6 @@ int mshv_set_msrs(const CPUState *cpu)int ret;size_t i, j;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);
@@ -449,7 +446,7 @@ int mshv_set_msrs(const CPUState *cpu) }/* SINT MSRs can only be written if SCONTROL has been set, so we split */ -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {
target/i386/mshv/synic.c new file +155 -0
155 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 07/12] target/i386/mshv: migrate SIMP and SIEFP state
Quoted 414 lines · Magnus Kulke
This part SynIC state is retrieved from the hypervisor via aligned state pages:
- Add new synic source file
- Centralize thesynic_enabled()check
- r/w pages from the hyper via aligned pages
- only handle pages when synic is enabled
- add buffers for migration to VM stateSigned-off-by Magnus Kulke <[email protected]>--- include/system/mshv_int.h | 7 ++ target/i386/cpu.h | 5 ++ target/i386/machine.c | 26 ++++++ target/i386/mshv/meson.build | 1 + target/i386/mshv/mshv-cpu.c | 64 +++++++++++++++ target/i386/mshv/msr.c | 7 +- target/i386/mshv/synic.c | 155 +++++++++++++++++++++++++++++++++++ 7 files changed, 260 insertions(+), 5 deletions(-) create mode 100644 target/i386/mshv/synic.cinclude/system/mshv_int.h +7
-0@@ -138,4 +138,11 @@ int mshv_init_msrs(const CPUState *cpu);int mshv_get_msrs (CPUState *cpu);int mshv_set_msrs (const CPUState *cpu); +/* synic */ +int mshv_get_simp (int cpu_fd,uint8_t *page); +int mshv_set_simp (int cpu_fd,const uint8_t *page); +int mshv_get_siefp (int cpu_fd,uint8_t *page); +int mshv_set_siefp (int cpu_fd,const uint8_t *page); +bool mshv_synic_enabled (const CPUState *cpu); +#endif target/i386/cpu.h +5
-0@@ -33,6 +33,7 @@#include "qemu/cpu-float.h" #include "qemu/timer.h" #include "standard-headers/asm-x86/kvm_para.h" +#include "hw/hyperv/hvgdk_mini.h" #define XEN_NR_VIRQS 24@@ -2299,6 +2300,10 @@ typedef struct CPUArchState {#if defined(CONFIG_HVF ) || defined(CONFIG_MSHV ) || defined(CONFIG_WHPX )void *emu_mmio_buf;#endif +#if defined(CONFIG_MSHV ) +uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +#endif uint64_t mcg_cap;uint64_t mcg_ctl;target/i386/machine.c +26
-0@@ -951,6 +951,29 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { } }; +#ifdef CONFIG_MSHV +static bool mshv_synic_vp_state_needed (void *opaque) +{ +X86CPU *cpu = opaque; +CPUX86State *env = &cpu->env; + +/* Only migrate SIMP/SIEFP if SynIC is enabled */ +return env->msr_hv_synic_control &1 ; +} + +static const VMStateDescription vmstate_mshv_synic_vp_state = { + .name ="cpu/mshv_synic_vp_state" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synic_vp_state_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_simp_page,X86CPU ), +VMSTATE_BUFFER (env.hv_siefp_page,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +#endif +static bool avx512_needed (void *opaque) {X86CPU *cpu = opaque;@@ -1915,6 +1938,9 @@ const VMStateDescription vmstate_x86_cpu = { &vmstate_cet,#ifdef TARGET_X86_64 &vmstate_apx, +#endif +#ifdef CONFIG_MSHV + &vmstate_mshv_synic_vp_state, #endifNULL }target/i386/mshv/meson.build +1
-0@@ -3,6 +3,7 @@ i386_mshv_ss = ss.source_set() i386_mshv_ss.add(files( 'mshv-cpu.c', 'msr.c', + 'synic.c', )) i386_system_ss.add_all(when: 'CONFIG_MSHV', if_true: i386_mshv_ss)target/i386/mshv/mshv-cpu.c +64
-0@@ -111,6 +111,33 @@ static enum hv_register_name FPU_REGISTER_NAMES[26] = {static int set_special_regs (const CPUState *cpu); +static int get_synic_state (CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); +CPUX86State *env = &x86cpu->env; +int cpu_fd =mshv_vcpufd (cpu); +int ret; + +/* SIMP/SIEFP can only be read when SynIC is enabled */ +if (!mshv_synic_enabled (cpu)) { +return 0 ; + } + + ret =mshv_get_simp (cpu_fd, env->hv_simp_page); +if (ret <0 ) { +error_report ("failed to get simp state" ); +return -1 ; + } + + ret =mshv_get_siefp (cpu_fd, env->hv_siefp_page); +if (ret <0 ) { +error_report ("failed to get siefp state" ); +return -1 ; + } + +return 0 ; +} +static int get_xsave_state (CPUState *cpu) {X86CPU *x86cpu =X86_CPU (cpu);@@ -969,6 +996,11 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } + ret =get_synic_state (cpu); +if (ret <0 ) { +return ret; + } + ret =get_vcpu_events (cpu);if (ret <0 ) {return ret;@@ -1381,6 +1413,33 @@ static int set_xc_reg(const CPUState *cpu)return 0 ; } +static int set_synic_state (const CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); +CPUX86State *env = &x86cpu->env; +int cpu_fd =mshv_vcpufd (cpu); +int ret; + +/* SIMP/SIEFP can only be written when SynIC is enabled */ +if (!mshv_synic_enabled (cpu)) { +return 0 ; + } + + ret =mshv_set_simp (cpu_fd, env->hv_simp_page); +if (ret <0 ) { +error_report ("failed to set simp state" ); +return -1 ; + } + + ret =mshv_set_siefp (cpu_fd, env->hv_siefp_page); +if (ret <0 ) { +error_report ("failed to set siefp state" ); +return -1 ; + } + +return 0 ; +} +int mshv_arch_store_vcpu_state (const CPUState *cpu) {int ret;@@ -1421,6 +1480,11 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } + ret =set_synic_state (cpu); +if (ret <0 ) { +return ret; + } + ret =set_vcpu_events (cpu);if (ret <0 ) {return ret;target/i386/mshv/msr.c +2
-5@@ -332,7 +332,6 @@ int mshv_get_msrs(CPUState *cpu)size_t i, j;uint32_t name;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled;set_hv_name_in_assocs (assocs, n_assocs);@@ -360,8 +359,7 @@ int mshv_get_msrs(CPUState *cpu)store_in_env (cpu, assocs, n_assocs);/* Read SINT MSRs only if SynIC is enabled */ - synic_enabled = x86cpu->env.msr_hv_synic_control &1 ; -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {@@ -415,7 +413,6 @@ int mshv_set_msrs(const CPUState *cpu)int ret;size_t i, j;X86CPU *x86cpu =X86_CPU (cpu); -bool synic_enabled = x86cpu->env.msr_hv_synic_control &1 ;load_from_env (cpu, assocs, n_assocs);@@ -449,7 +446,7 @@ int mshv_set_msrs(const CPUState *cpu) }/* SINT MSRs can only be written if SCONTROL has been set, so we split */ -if (synic_enabled) { +if (mshv_synic_enabled (cpu)) {QEMU_BUILD_BUG_ON (MSHV_MSR_TOTAL_COUNT <HV_SINT_COUNT );for (i =0 ; i <HV_SINT_COUNT ; i++) {target/i386/mshv/synic.c new file +155
-0@@ -0,0 +1,155 @@ +/* +* QEMU MSHV SynIC support +* +* Copyright Microsoft, Corp. 2026 +* +* Authors: Magnus Kulke <[email protected]> +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ + +#include "qemu/osdep.h" +#include "qemu/memalign.h" +#include "qemu/error-report.h" + +#include "system/mshv.h" +#include "system/mshv_int.h" + +#include "linux/mshv.h" +#include "hw/hyperv/hvgdk_mini.h" +#include "cpu.h" + +#include <sys/ioctl.h> + +bool mshv_synic_enabled (const CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); + +return x86cpu->env.msr_hv_synic_control &1 ; +} + +static int get_vp_state (int cpu_fd,struct mshv_get_set_vp_state *state) +{ +int ret; + + ret =ioctl (cpu_fd,MSHV_GET_VP_STATE , state); +if (ret <0 ) { +error_report ("failed to get vp state: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +static int set_vp_state (int cpu_fd,const struct mshv_get_set_vp_state *state) +{ +int ret; + + ret =ioctl (cpu_fd,MSHV_SET_VP_STATE , state); +if (ret <0 ) { +error_report ("failed to set vp state: %s" ,strerror (errno)); +return -1 ; + } + +return 0 ; +} + +int mshv_get_simp (int cpu_fd,uint8_t *page) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SIMP ; + + ret =get_vp_state (cpu_fd, &args); + +if (ret <0 ) { +qemu_vfree (buffer); +error_report ("failed to get simp" ); +return -1 ; + } + +memcpy (page, buffer,HV_HYP_PAGE_SIZE ); +qemu_vfree (buffer); + +return 0 ; +} + +int mshv_set_simp (int cpu_fd,const uint8_t *page) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SIMP ; + +assert (page); +memcpy (buffer, page,HV_HYP_PAGE_SIZE ); + + ret =set_vp_state (cpu_fd, &args); +qemu_vfree (buffer); + +if (ret <0 ) { +error_report ("failed to set simp" ); +return -1 ; + } + +return 0 ; +} + +int mshv_get_siefp (int cpu_fd,uint8_t *page) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SIEFP , + + ret =get_vp_state (cpu_fd, &args); + +if (ret <0 ) { +qemu_vfree (buffer); +error_report ("failed to get siefp" ); +return -1 ; + } + +memcpy (page, buffer,HV_HYP_PAGE_SIZE ); +qemu_vfree (buffer); + +return 0 ; +} + +int mshv_set_siefp (int cpu_fd,const uint8_t *page) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SIEFP , + +assert (page); +memcpy (buffer, page,HV_HYP_PAGE_SIZE ); + + ret =set_vp_state (cpu_fd, &args); +qemu_vfree (buffer); + +if (ret <0 ) { +error_report ("failed to set simp" ); +return -1 ; + } + +return 0 ; +}Signature · 1 line
2.34.1
PATCH 8/12 target/i386/mshv: migrate STIMER state
In reply to Magnus KulkeThis part of Synic state is retrieved via a mem-aligned page. We declare the required space (size reference: rust-vmm/mshv) as a buffer on the VM state struct for inclusion in a migration.
Other than other SynIC features, STIMER doesn't depend on SCONTROL being set.
--- include/system/mshv_int.h | 2 ++ target/i386/cpu.h | 5 ++++ target/i386/machine.c | 20 +++++++++++++++ target/i386/mshv/mshv-cpu.c | 12 +++++++++ target/i386/mshv/synic.c | 51 +++++++++++++++++++++++++++++++++++++ 5 files changed, 90 insertions(+)
include/system/mshv_int.h +2 -0
@@ -144,5 +144,7 @@ int mshv_set_simp(int cpu_fd, const uint8_t *page);int mshv_get_siefp (int cpu_fd,uint8_t *page);int mshv_set_siefp (int cpu_fd,const uint8_t *page);bool mshv_synic_enabled (const CPUState *cpu); +int mshv_get_synthetic_timers (int cpu_fd,uint8_t *state); +int mshv_set_synthetic_timers (int cpu_fd,const uint8_t *state);#endif
target/i386/cpu.h +5 -0
@@ -45,6 +45,10 @@#define ELF_MACHINE_UNAME "i686"#endif +#ifdef CONFIG_MSHV +#define MSHV_STIMERS_STATE_SIZE 200 +#endif +enum {R_EAX =0 ,R_ECX =1 ,
@@ -2303,6 +2307,7 @@ typedef struct CPUArchState {#if defined(CONFIG_MSHV )uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ];uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_synthetic_timers_state[MSHV_STIMERS_STATE_SIZE ];#endif uint64_t mcg_cap;
target/i386/machine.c +20 -0
@@ -10,6 +10,7 @@#include "exec/watchpoint.h" #include "system/kvm.h" #include "system/kvm_xen.h" +#include "system/mshv.h" #include "system/tcg.h" #include "qemu/error-report.h"
@@ -952,6 +953,24 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { };#ifdef CONFIG_MSHV + +static bool mshv_synthetic_timers_needed (void *opaque) +{ +/* Always migrate synthetic timers */ +return mshv_enabled (); +} + +static const VMStateDescription vmstate_mshv_synthetic_timers = { + .name ="cpu/mshv_synthetic_timers" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synthetic_timers_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_synthetic_timers_state,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +static bool mshv_synic_vp_state_needed (void *opaque) {X86CPU *cpu = opaque;
@@ -1941,6 +1960,7 @@ const VMStateDescription vmstate_x86_cpu = {#endif #ifdef CONFIG_MSHV &vmstate_mshv_synic_vp_state, + &vmstate_mshv_synthetic_timers, #endifNULL }
target/i386/mshv/mshv-cpu.c +12 -0
@@ -118,6 +118,12 @@ static int get_synic_state(CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_get_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to get synthetic timers" ); +return -1 ; + } +/* SIMP/SIEFP can only be read when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;
@@ -1420,6 +1426,12 @@ static int set_synic_state(const CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_set_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to set synthetic timers state" ); +return -1 ; + } +/* SIMP/SIEFP can only be written when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;
target/i386/mshv/synic.c +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 08/12] target/i386/mshv: migrate STIMER state
Quoted 190 lines · Magnus Kulke
This part of Synic state is retrieved via a mem-aligned page. We declare the required space (size reference: rust-vmm/mshv) as a buffer on the VM state struct for inclusion in a migration.
Other than other SynIC features, STIMER doesn't depend on SCONTROL being set.
Signed-off-by Magnus Kulke <[email protected]>--- include/system/mshv_int.h | 2 ++ target/i386/cpu.h | 5 ++++ target/i386/machine.c | 20 +++++++++++++++ target/i386/mshv/mshv-cpu.c | 12 +++++++++ target/i386/mshv/synic.c | 51 +++++++++++++++++++++++++++++++++++++ 5 files changed, 90 insertions(+)include/system/mshv_int.h +2
-0@@ -144,5 +144,7 @@ int mshv_set_simp(int cpu_fd, const uint8_t *page);int mshv_get_siefp (int cpu_fd,uint8_t *page);int mshv_set_siefp (int cpu_fd,const uint8_t *page);bool mshv_synic_enabled (const CPUState *cpu); +int mshv_get_synthetic_timers (int cpu_fd,uint8_t *state); +int mshv_set_synthetic_timers (int cpu_fd,const uint8_t *state);#endif target/i386/cpu.h +5
-0@@ -45,6 +45,10 @@#define ELF_MACHINE_UNAME "i686"#endif +#ifdef CONFIG_MSHV +#define MSHV_STIMERS_STATE_SIZE 200 +#endif +enum {R_EAX =0 ,R_ECX =1 ,@@ -2303,6 +2307,7 @@ typedef struct CPUArchState {#if defined(CONFIG_MSHV )uint8_t hv_simp_page[HV_HYP_PAGE_SIZE ];uint8_t hv_siefp_page[HV_HYP_PAGE_SIZE ]; +uint8_t hv_synthetic_timers_state[MSHV_STIMERS_STATE_SIZE ];#endif uint64_t mcg_cap;target/i386/machine.c +20
-0@@ -10,6 +10,7 @@#include "exec/watchpoint.h" #include "system/kvm.h" #include "system/kvm_xen.h" +#include "system/mshv.h" #include "system/tcg.h" #include "qemu/error-report.h" @@ -952,6 +953,24 @@ static const VMStateDescription vmstate_msr_hyperv_reenlightenment = { };#ifdef CONFIG_MSHV + +static bool mshv_synthetic_timers_needed (void *opaque) +{ +/* Always migrate synthetic timers */ +return mshv_enabled (); +} + +static const VMStateDescription vmstate_mshv_synthetic_timers = { + .name ="cpu/mshv_synthetic_timers" , + .version_id =1 , + .minimum_version_id =1 , + .needed = mshv_synthetic_timers_needed, + .fields = (const VMStateField []) { +VMSTATE_BUFFER (env.hv_synthetic_timers_state,X86CPU ), +VMSTATE_END_OF_LIST () + } +}; +static bool mshv_synic_vp_state_needed (void *opaque) {X86CPU *cpu = opaque;@@ -1941,6 +1960,7 @@ const VMStateDescription vmstate_x86_cpu = {#endif #ifdef CONFIG_MSHV &vmstate_mshv_synic_vp_state, + &vmstate_mshv_synthetic_timers, #endifNULL }target/i386/mshv/mshv-cpu.c +12
-0@@ -118,6 +118,12 @@ static int get_synic_state(CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_get_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to get synthetic timers" ); +return -1 ; + } +/* SIMP/SIEFP can only be read when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;@@ -1420,6 +1426,12 @@ static int set_synic_state(const CPUState *cpu)int cpu_fd =mshv_vcpufd (cpu);int ret; + ret =mshv_set_synthetic_timers (cpu_fd, env->hv_synthetic_timers_state); +if (ret <0 ) { +error_report ("failed to set synthetic timers state" ); +return -1 ; + } +/* SIMP/SIEFP can only be written when SynIC is enabled */ if (!mshv_synic_enabled (cpu)) {return 0 ;target/i386/mshv/synic.c +51
-0@@ -54,6 +54,57 @@ static int set_vp_state(int cpu_fd, const struct mshv_get_set_vp_state *state)return 0 ; } +int mshv_get_synthetic_timers (int cpu_fd,uint8_t *state) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SYNTHETIC_TIMERS ; + + ret =get_vp_state (cpu_fd, &args); + +if (ret <0 ) { +qemu_vfree (buffer); +error_report ("failed to get synthetic timers" ); +return -1 ; + } + +memcpy (state, buffer,MSHV_STIMERS_STATE_SIZE ); +qemu_vfree (buffer); + +return 0 ; +} + +int mshv_set_synthetic_timers (int cpu_fd,const uint8_t *state) +{ +int ret; +void *buffer; +struct mshv_get_set_vp_state args = {0 }; + + buffer =qemu_memalign (HV_HYP_PAGE_SIZE ,HV_HYP_PAGE_SIZE ); +memset (buffer,0 ,HV_HYP_PAGE_SIZE ); + args.buf_ptr = (uint64_t )buffer; + args.buf_sz =HV_HYP_PAGE_SIZE ; + args.type =MSHV_VP_STATE_SYNTHETIC_TIMERS ; + +assert (state); +memcpy (buffer, state,MSHV_STIMERS_STATE_SIZE ); + + ret =set_vp_state (cpu_fd, &args); +qemu_vfree (buffer); + +if (ret <0 ) { +error_report ("failed to set synthetic timers" ); +return -1 ; + } + +return 0 ; +} +int mshv_get_simp (int cpu_fd,uint8_t *page) {int ret;Signature · 1 line
2.34.1
PATCH 9/12 accel/mshv: write synthetic MSRs after migration
In reply to Magnus KulkeWrite partition-wide synthetic MSRs. This ensures the hypercall page and SynIC facilities are set up before vCPUs attempt to use it.
--- accel/mshv/mshv-all.c | 15 +++++++++++++++ include/hw/hyperv/hvgdk_mini.h | 3 +++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 15 +++++++++++++++ 4 files changed, 34 insertions(+)
accel/mshv/mshv-all.c +15 -0
@@ -60,6 +60,20 @@ static int init_mshv(int *mshv_fd)return 0 ; } +static int mshv_load_cleanup (void *opaque) +{ +int ret; + + ret =mshv_arch_set_partition_msrs (first_cpu); +if (ret <0 ) { +error_report ("Failed to set partition MSRs: %s" ,strerror (-ret)); +return -1 ; + } + +return 0 ; +} + +static int get_host_partition_property (int mshv_fd,uint32_t property_code,uint64_t *value) {
@@ -496,6 +510,7 @@ static int mshv_init_vcpu(CPUState *cpu) }static SaveVMHandlers savevm_mshv = { + .load_cleanup = mshv_load_cleanup, };static int mshv_init (AccelState *as,MachineState *ms)
include/hw/hyperv/hvgdk_mini.h +3 -0
@@ -23,6 +23,9 @@#define HV_X64_MSR_APIC_FREQUENCY 0x40000023 typedef enum hv_register_name { + /* VP Management Registers */ + HV_REGISTER_INTERNAL_ACTIVITY_STATE = 0x00000004, + /* Pending Interruption Register */ HV_REGISTER_PENDING_INTERRUPTION = 0x00010002, HV_REGISTER_INTERRUPT_STATE = 0x00010003,
include/system/mshv_int.h +1 -0
@@ -98,6 +98,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs, size_t n_regs);int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu); +int mshv_arch_set_partition_msrs (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +15 -0
@@ -1505,6 +1505,21 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return 0 ; } +int mshv_arch_set_partition_msrs (const CPUState *cpu) +{ +CPUX86State *env = &X86_CPU (cpu)->env; +struct hv_register_assoc assocs[] = { + { .name =HV_REGISTER_GUEST_OS_ID , + .value.reg64 = env->msr_hv_guest_os_id }, + { .name =HV_REGISTER_REFERENCE_TSC , + .value.reg64 = env->msr_hv_tsc }, + { .name =HV_X64_REGISTER_HYPERCALL , + .value.reg64 = env->msr_hv_hypercall }, + }; + +return mshv_set_generic_regs (cpu, assocs,ARRAY_SIZE (assocs)); +} +void mshv_arch_amend_proc_features (union hv_partition_synthetic_processor_features *features) {
Signature · 1 line
2.34.1
Re: [PATCH 09/12] accel/mshv: write synthetic MSRs after migration
Quoted 98 lines · Magnus Kulke
Write partition-wide synthetic MSRs. This ensures the hypercall page and SynIC facilities are set up before vCPUs attempt to use it.
Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 15 +++++++++++++++ include/hw/hyperv/hvgdk_mini.h | 3 +++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 15 +++++++++++++++ 4 files changed, 34 insertions(+)accel/mshv/mshv-all.c +15
-0@@ -60,6 +60,20 @@ static int init_mshv(int *mshv_fd)return 0 ; } +static int mshv_load_cleanup (void *opaque) +{ +int ret; + + ret =mshv_arch_set_partition_msrs (first_cpu); +if (ret <0 ) { +error_report ("Failed to set partition MSRs: %s" ,strerror (-ret)); +return -1 ; + } + +return 0 ; +} + +static int get_host_partition_property (int mshv_fd,uint32_t property_code,uint64_t *value) {@@ -496,6 +510,7 @@ static int mshv_init_vcpu(CPUState *cpu) }static SaveVMHandlers savevm_mshv = { + .load_cleanup = mshv_load_cleanup, };static int mshv_init (AccelState *as,MachineState *ms)include/hw/hyperv/hvgdk_mini.h +3
-0@@ -23,6 +23,9 @@#define HV_X64_MSR_APIC_FREQUENCY 0x40000023 typedef enum hv_register_name { + /* VP Management Registers */ + HV_REGISTER_INTERNAL_ACTIVITY_STATE = 0x00000004, + /* Pending Interruption Register */ HV_REGISTER_PENDING_INTERRUPTION = 0x00010002, HV_REGISTER_INTERRUPT_STATE = 0x00010003,include/system/mshv_int.h +1
-0@@ -98,6 +98,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs, size_t n_regs);int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu); +int mshv_arch_set_partition_msrs (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(target/i386/mshv/mshv-cpu.c +15
-0@@ -1505,6 +1505,21 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return 0 ; } +int mshv_arch_set_partition_msrs (const CPUState *cpu) +{ +CPUX86State *env = &X86_CPU (cpu)->env; +struct hv_register_assoc assocs[] = { + { .name =HV_REGISTER_GUEST_OS_ID , + .value.reg64 = env->msr_hv_guest_os_id }, + { .name =HV_REGISTER_REFERENCE_TSC , + .value.reg64 = env->msr_hv_tsc }, + { .name =HV_X64_REGISTER_HYPERCALL , + .value.reg64 = env->msr_hv_hypercall }, + }; + +return mshv_set_generic_regs (cpu, assocs,ARRAY_SIZE (assocs)); +} +void mshv_arch_amend_proc_features (union hv_partition_synthetic_processor_features *features) {Signature · 1 line
2.34.1
PATCH 10/12 target/i386/mshv: migrate MP_STATE
In reply to Magnus KulkeMSHV's "internal activity state" roughly maps to QEMU's env->mp_state and cpu->halted states that describe state of APs in a guest.
We don't invoke set_mp_state as part of store_vcpu_state() b/c we would put all BSP + APs in a RUNNABLE (0) state immediately, breaking SMP boot
Instead we store the mp state as part of the load_cleanup() routine after a migration.
--- accel/mshv/mshv-all.c | 10 +++++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 80 +++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+)
accel/mshv/mshv-all.c +10 -0
@@ -62,6 +62,7 @@ static int init_mshv(int *mshv_fd)static int mshv_load_cleanup (void *opaque) { +CPUState *cpu;int ret; ret =mshv_arch_set_partition_msrs (first_cpu);
@@ -70,6 +71,15 @@ static int mshv_load_cleanup(void *opaque)return -1 ; } +CPU_FOREACH (cpu) { + ret =mshv_arch_set_mp_state (cpu); +if (ret <0 ) { +error_report ("Failed to set mp state for vCPU %d: %s" , + cpu->cpu_index,strerror (-ret)); +return -1 ; + } + } +return 0 ; }
include/system/mshv_int.h +1 -0
@@ -99,6 +99,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu);int mshv_arch_set_partition_msrs (const CPUState *cpu); +int mshv_arch_set_mp_state (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(
target/i386/mshv/mshv-cpu.c +80 -0
98 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH 10/12] target/i386/mshv: migrate MP_STATE
Quoted 163 lines · Magnus Kulke
MSHV's "internal activity state" roughly maps to QEMU's
env->mp_stateandcpu->haltedstates that describe state of APs in a guest.We don't invoke
set_mp_stateas part ofstore_vcpu_state()b/c we would put all BSP + APs in a RUNNABLE (0) state immediately, breaking SMP bootInstead we store the mp state as part of the
load_cleanup()routine after a migration.Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 10 +++++ include/system/mshv_int.h | 1 + target/i386/mshv/mshv-cpu.c | 80 +++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+)accel/mshv/mshv-all.c +10
-0@@ -62,6 +62,7 @@ static int init_mshv(int *mshv_fd)static int mshv_load_cleanup (void *opaque) { +CPUState *cpu;int ret; ret =mshv_arch_set_partition_msrs (first_cpu);@@ -70,6 +71,15 @@ static int mshv_load_cleanup(void *opaque)return -1 ; } +CPU_FOREACH (cpu) { + ret =mshv_arch_set_mp_state (cpu); +if (ret <0 ) { +error_report ("Failed to set mp state for vCPU %d: %s" , + cpu->cpu_index,strerror (-ret)); +return -1 ; + } + } +return 0 ; }include/system/mshv_int.h +1
-0@@ -99,6 +99,7 @@ int mshv_get_generic_regs(CPUState *cpu, hv_register_assoc *assocs,int mshv_arch_store_vcpu_state (const CPUState *cpu);int mshv_arch_load_vcpu_state (CPUState *cpu);int mshv_arch_set_partition_msrs (const CPUState *cpu); +int mshv_arch_set_mp_state (const CPUState *cpu);void mshv_arch_init_vcpu (CPUState *cpu);void mshv_arch_destroy_vcpu (CPUState *cpu); void mshv_arch_amend_proc_features(target/i386/mshv/mshv-cpu.c +80
-0@@ -35,6 +35,11 @@#include <sys/ioctl.h> +#define MSHV_MP_STATE_RUNNABLE 0 +#define MSHV_MP_STATE_UNINITIALIZED 1 +#define MSHV_MP_STATE_INIT_RECEIVED 2 +#define MSHV_MP_STATE_HALTED 3 +#define MAX_REGISTER_COUNT (MAX_CONST(ARRAY_SIZE(STANDARD_REGISTER_NAMES), \ MAX_CONST(ARRAY_SIZE(SPECIAL_REGISTER_NAMES), \ ARRAY_SIZE(FPU_REGISTER_NAMES))))@@ -950,6 +955,76 @@ static int set_vcpu_events(const CPUState *cpu)return 0 ; } +static int get_mp_state (CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); +CPUX86State *env = &x86cpu->env; +struct hv_register_assoc assoc = { + .name =HV_REGISTER_INTERNAL_ACTIVITY_STATE , + }; +union hv_internal_activity_register activity; +int ret; + + ret =mshv_get_generic_regs (cpu, &assoc,1 ); +if (ret <0 ) { +error_report ("failed to get internal activity state" ); +return -1 ; + } + + activity.as_uint64 = assoc.value.reg64; + +/* +* map MSHV activity state to KVM mp_state values, which are used as the +* shared representation in env->mp_state and serialized by vmstate_x86_cpu. +*/ + +if (activity.startup_suspend) { + env->mp_state =MSHV_MP_STATE_UNINITIALIZED ; + }else if (activity.halt_suspend) { + env->mp_state =MSHV_MP_STATE_HALTED ; + }else { + env->mp_state =MSHV_MP_STATE_RUNNABLE ; + } + + cpu->halted = (env->mp_state ==MSHV_MP_STATE_HALTED ); + +return 0 ; +} + +int mshv_arch_set_mp_state (const CPUState *cpu) +{ +X86CPU *x86cpu =X86_CPU (cpu); +CPUX86State *env = &x86cpu->env; +union hv_internal_activity_register activity = {0 }; +struct hv_register_assoc assoc = { + .name =HV_REGISTER_INTERNAL_ACTIVITY_STATE , + }; +int ret; + +switch (env->mp_state) { +case MSHV_MP_STATE_HALTED : + activity.halt_suspend =1 ; +break ; +case MSHV_MP_STATE_UNINITIALIZED : +case MSHV_MP_STATE_INIT_RECEIVED : + activity.startup_suspend =1 ; +break ; +case MSHV_MP_STATE_RUNNABLE : +default : +break ; + } + + assoc.value.reg64 = activity.as_uint64; + + ret =mshv_set_generic_regs (cpu, &assoc,1 ); +if (ret <0 ) { +error_report ("failed to set internal activity state" ); +return -1 ; + } + +return 0 ; +} +static int update_hflags (CPUState *cpu) {X86CPU *x86cpu =X86_CPU (cpu);@@ -1012,6 +1087,11 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } + ret =get_mp_state (cpu); +if (ret <0 ) { +return ret; + } +return 0 ; }Signature · 1 line
2.34.1
PATCH 11/12 target/i386/mshv: toggle fpu/xsave migration
In reply to Magnus KulkeMSHV exposes overlapping legacy FP/SSE state through two paths:
explicit Hyper-V FPU/XMM + registers and VP XSAVE state. There can be subtle inconsistencies across migrations when XSAVE is written after FPU state.
--- target/i386/mshv/mshv-cpu.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
target/i386/mshv/mshv-cpu.c +5 -4
@@ -1057,7 +1057,7 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } - ret =get_fpu (cpu); + ret =get_xsave_state (cpu);if (ret <0 ) {return ret; }
@@ -1072,7 +1072,7 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } - ret =get_xsave_state (cpu); + ret =get_fpu (cpu);if (ret <0 ) {return ret; }
@@ -1551,7 +1551,7 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } - ret =set_fpu (cpu); + ret =set_xsave_state (cpu);if (ret <0 ) {return ret; }
@@ -1567,7 +1567,8 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } - ret =set_xsave_state (cpu); +/* INVARIANT: legacy FPU state must be restored after XSAVE */ + ret =set_fpu (cpu);if (ret <0 ) {return ret; }
Signature · 1 line
2.34.1
Re: [PATCH 11/12] target/i386/mshv: toggle fpu/xsave migration
Quoted 53 lines · Magnus Kulke
MSHV exposes overlapping legacy FP/SSE state through two paths:
explicit Hyper-V FPU/XMM + registers and VP XSAVE state. There can be subtle inconsistencies across migrations when XSAVE is written after FPU state.Signed-off-by Magnus Kulke <[email protected]>--- target/i386/mshv/mshv-cpu.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)target/i386/mshv/mshv-cpu.c +5
-4@@ -1057,7 +1057,7 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } - ret =get_fpu (cpu); + ret =get_xsave_state (cpu);if (ret <0 ) {return ret; }@@ -1072,7 +1072,7 @@ int mshv_arch_load_vcpu_state(CPUState *cpu)return ret; } - ret =get_xsave_state (cpu); + ret =get_fpu (cpu);if (ret <0 ) {return ret; }@@ -1551,7 +1551,7 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } - ret =set_fpu (cpu); + ret =set_xsave_state (cpu);if (ret <0 ) {return ret; }@@ -1567,7 +1567,8 @@ int mshv_arch_store_vcpu_state(const CPUState *cpu)return ret; } - ret =set_xsave_state (cpu); +/* INVARIANT: legacy FPU state must be restored after XSAVE */ + ret =set_fpu (cpu);if (ret <0 ) {return ret; }Signature · 1 line
2.34.1
PATCH 12/12 hw/i386/mshv: drop initial msi vector 0
In reply to Magnus KulkeThis has been a warning before that was always raised if the machine has a hpet. hpet_reset() will eventually result in mshv_send_msi called w/ vector 0, which we can safely drop.
--- hw/i386/mshv/apic.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
hw/i386/mshv/apic.c +10 -0
@@ -293,6 +293,16 @@ static void mshv_send_msi(MSIMessage *msi) delivery = (data >>MSI_DATA_DELIVERY_MODE_SHIFT ) &MSI_DATA_DELIVERY_MODE_MASK ; +/* +* Vector 0 is not a valid interrupt vector (0-15 are reserved for CPU +* exceptions). This can trigger during machine reset, if hpet_reset() +* forces the PIT to pulse GSI 2 before IOAPIC's own reset has masked its +* redirection entries. +*/ +if (vector ==0 ) { +return ; + } +mshv_request_interrupt (mshv_state, delivery, vector, dest, dest_mode, trigger_mode); }
Signature · 1 line
2.34.1
Re: [PATCH 12/12] hw/i386/mshv: drop initial msi vector 0
Quoted 32 lines · Magnus Kulke
This has been a warning before that was always raised if the machine has a hpet.
hpet_reset()will eventually result inmshv_send_msicalled w/ vector 0, which we can safely drop.Signed-off-by Magnus Kulke <[email protected]>--- hw/i386/mshv/apic.c | 10 ++++++++++ 1 file changed, 10 insertions(+)hw/i386/mshv/apic.c +10
-0@@ -293,6 +293,16 @@ static void mshv_send_msi(MSIMessage *msi) delivery = (data >>MSI_DATA_DELIVERY_MODE_SHIFT ) &MSI_DATA_DELIVERY_MODE_MASK ; +/* +* Vector 0 is not a valid interrupt vector (0-15 are reserved for CPU +* exceptions). This can trigger during machine reset, if hpet_reset() +* forces the PIT to pulse GSI 2 before IOAPIC's own reset has masked its +* redirection entries. +*/ +if (vector ==0 ) { +return ; + } +mshv_request_interrupt (mshv_state, delivery, vector, dest, dest_mode, trigger_mode); }Signature · 1 line
2.34.1