Support QEMU cpu models in MSHV accelerator
v2 cover Support QEMU cpu models in MSHV accelerator
Hey all,
In the current MSHV accelerator code passing CPU features via the -cpu flag doesn't work as intended yet. When using the MSHV hypervisor we either silently discard the specified model/features and leave it up to the hypervisor to provide a sensible set of features or if the user selects -cpu host, the hypervisor might refuse to create a partition.
This changeset introduces a more comprehensive support for passing desired guest cpu features to the hypervisor. It's also a prerequisite for Live Migration support.
Drive-by fixes:
- adding packed attribute to inlined UAPI structs
Changes since v1:
- query hypervisor for supported processor features instead of the host OS
- disable la57 (5 level page tables)
best,
magnus
Magnus Kulke (9):
accel/mshv: use mshv_create_partition_v2 payload target/i386/mshv: fix cpuid propagation bug target/i386/mshv: fix various cpuid traversal bugs target/i386/mshv: change cpuid mask to UINT32_MAX target/i386/mshv: set cpu model name on -cpu host target/i386: query mshv accel for supported cpuids target/i386/mshv: populate xsave area offsets target/i386/mshv: filter out CET bits in cpuid accel/mshv: disable la57 (5lvl paging)
accel/mshv/mshv-all.c | 82 +++++++++++- include/hw/hyperv/hvgdk_mini.h | 2 + include/hw/hyperv/hvhdk.h | 195 ++++++++++++++++++++++++++++ include/system/mshv.h | 3 + target/i386/cpu.c | 8 ++ target/i386/mshv/mshv-cpu.c | 227 +++++++++++++++++++++++++++------ 6 files changed, 468 insertions(+), 49 deletions(-)
Signature · 1 line
2.34.1
PATCH v2 1/9 accel/mshv: use mshv_create_partition_v2 payload
When using the extended request format we can instruct the hypervisor to provision enough space for requested XSAVE features. This is required for supporting QEMU models provided via the -cpu flag properly.
--- accel/mshv/mshv-all.c | 75 +++++++++++-- include/hw/hyperv/hvgdk_mini.h | 2 + include/hw/hyperv/hvhdk.h | 195 +++++++++++++++++++++++++++++++++ 3 files changed, 265 insertions(+), 7 deletions(-)
accel/mshv/mshv-all.c +68 -7
@@ -110,21 +110,82 @@ static int resume_vm(int vm_fd)return 0 ; } +static int get_host_partition_property (int mshv_fd,uint32_t feature_bank, +uint64_t *value) +{ +int ret; +struct hv_input_get_partition_property in = {0 }; +struct hv_output_get_partition_property out = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code = feature_bank; + + args.code =HVCALL_GET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + args.out_sz =sizeof (out); + args.out_ptr = (uint64_t )&out; + + ret =ioctl (mshv_fd,MSHV_ROOT_HVCALL , &args); +if (ret <0 ) { +error_report ("Failed to get host partition property bank: %s" , +strerror (errno)); +return -1 ; + } + + *value = out.property_value; +return 0 ; +} +static int create_partition (int mshv_fd,int *vm_fd) {int ret; -struct mshv_create_partition args = {0 }; +uint64_t pt_flags, host_proc_features; +union hv_partition_processor_xsave_features disabled_xsave_features; +struct mshv_create_partition_v2 args = {0 }; +uint32_t feature_banks[] = { +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 , +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1 , + }; + +QEMU_BUILD_BUG_ON (ARRAY_SIZE (feature_banks) >MSHV_NUM_CPU_FEATURES_BANKS );/* Initialize pt_flags with the desired features */ -uint64_t pt_flags = (1ULL <<MSHV_PT_BIT_LAPIC ) | - (1ULL <<MSHV_PT_BIT_X2APIC ) | - (1ULL <<MSHV_PT_BIT_GPA_SUPER_PAGES ); + pt_flags = (1ULL <<MSHV_PT_BIT_LAPIC ) | + (1ULL <<MSHV_PT_BIT_X2APIC ) | + (1ULL <<MSHV_PT_BIT_GPA_SUPER_PAGES ) | + (1ULL <<MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES ); + +/* enable all */ + disabled_xsave_features.as_uint64 =0 ; + +/* +* query host for supported processor features and disable unsupported +* features: (0 means supported, 1 means disabled, hence the negation) +*/ + ret =get_host_partition_property (mshv_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 , + &host_proc_features); +if (ret <0 ) { +error_report ("Failed to get host processor feature bank 0" ); +return -1 ; + } + args.pt_cpu_fbanks[0 ] = ~host_proc_features; -/* Set default isolation type */ -uint64_t pt_isolation =MSHV_PT_ISOLATION_NONE ; + ret =get_host_partition_property (mshv_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1 , + &host_proc_features); +if (ret <0 ) { +error_report ("Failed to get host processor feature bank 1" ); +return -1 ; + } + args.pt_cpu_fbanks[1 ] = ~host_proc_features; +/* populate args structure */ args.pt_flags = pt_flags; - args.pt_isolation = pt_isolation; + args.pt_isolation =MSHV_PT_ISOLATION_NONE ; + args.pt_disabled_xsave = disabled_xsave_features.as_uint64; + args.pt_num_cpu_fbanks =MSHV_NUM_CPU_FEATURES_BANKS ; ret =ioctl (mshv_fd,MSHV_CREATE_PARTITION , &args);if (ret <0 ) {
include/hw/hyperv/hvgdk_mini.h +2 -0
@@ -456,6 +456,8 @@ typedef struct hv_input_set_vp_registers {struct hv_register_assoc elements[]; }QEMU_PACKED hv_input_set_vp_registers; +#define MSHV_VP_MAX_REGISTERS 128 +union hv_interrupt_control {uint64_t as_uint64;struct {
include/hw/hyperv/hvhdk.h +195 -0
207 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH v2 1/9] accel/mshv: use mshv_create_partition_v2 payload
Quoted 5 lines · Magnus Kulke
When using the extended request format we can instruct the hypervisor to provision enough space for requested XSAVE features. This is required for supporting QEMU models provided via the -cpu flag properly.
Signed-off-by Magnus Kulke <[email protected]>
Re: [PATCH v2 1/9] accel/mshv: use mshv_create_partition_v2 payload
In reply to Magnus KulkeQuoted 16 lines · Magnus Kulke
When using the extended request format we can instruct the hypervisor to provision enough space for requested XSAVE features. This is required for supporting QEMU models provided via the -cpu flag properly.
Signed-off-by Magnus Kulke <[email protected]>--- accel/mshv/mshv-all.c | 75 +++++++++++-- include/hw/hyperv/hvgdk_mini.h | 2 + include/hw/hyperv/hvhdk.h | 195 +++++++++++++++++++++++++++++++++ 3 files changed, 265 insertions(+), 7 deletions(-)accel/mshv/mshv-all.c +0
-0
accel/mshv/mshv-all.c static int resume_vm(int vm_fd) · line 113
Quoted 4 lines
accel/mshv/mshv-all.c +1
-0@@ -110,21 +110,82 @@ static int resume_vm(int vm_fd)return 0 ; } +static int get_host_partition_property(int mshv_fd, uint32_t feature_bank,
Why not just call the param property_code instead of feature_bank? That way this function can be used for other properties too.
Quoted 29 lines
accel/mshv/mshv-all.c +26
-0+ uint64_t *value) +{ +int ret; +struct hv_input_get_partition_property in = {0 }; +struct hv_output_get_partition_property out = {0 }; +struct mshv_root_hvcall args = {0 }; + + in.property_code = feature_bank; + + args.code =HVCALL_GET_PARTITION_PROPERTY ; + args.in_sz =sizeof (in); + args.in_ptr = (uint64_t )∈ + args.out_sz =sizeof (out); + args.out_ptr = (uint64_t )&out; + + ret =ioctl (mshv_fd,MSHV_ROOT_HVCALL , &args); +if (ret <0 ) { +error_report ("Failed to get host partition property bank: %s" , +strerror (errno)); +return -1 ; + } + + *value = out.property_value; +return 0 ; +} +static int create_partition (int mshv_fd,int *vm_fd) {int ret;
accel/mshv/mshv-all.c static int resume_vm(int vm_fd) · line 151
Quoted 10 lines
accel/mshv/mshv-all.c +9
-1-struct mshv_create_partition args = {0 }; +uint64_t pt_flags, host_proc_features; +union hv_partition_processor_xsave_features disabled_xsave_features; +struct mshv_create_partition_v2 args = {0 }; +uint32_t feature_banks[] = { +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 , +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1 , + }; + +QEMU_BUILD_BUG_ON (ARRAY_SIZE (feature_banks) >MSHV_NUM_CPU_FEATURES_BANKS );
Is this really useful since we're not using the feature_banks array anywhere?
Thanks,
Anirudh.
Quoted 276 lines
accel/mshv/mshv-all.c +32
-6/* Initialize pt_flags with the desired features */ -uint64_t pt_flags = (1ULL <<MSHV_PT_BIT_LAPIC ) | - (1ULL <<MSHV_PT_BIT_X2APIC ) | - (1ULL <<MSHV_PT_BIT_GPA_SUPER_PAGES ); + pt_flags = (1ULL <<MSHV_PT_BIT_LAPIC ) | + (1ULL <<MSHV_PT_BIT_X2APIC ) | + (1ULL <<MSHV_PT_BIT_GPA_SUPER_PAGES ) | + (1ULL <<MSHV_PT_BIT_CPU_AND_XSAVE_FEATURES ); + +/* enable all */ + disabled_xsave_features.as_uint64 =0 ; + +/* +* query host for supported processor features and disable unsupported +* features: (0 means supported, 1 means disabled, hence the negation) +*/ + ret =get_host_partition_property (mshv_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 , + &host_proc_features); +if (ret <0 ) { +error_report ("Failed to get host processor feature bank 0" ); +return -1 ; + } + args.pt_cpu_fbanks[0 ] = ~host_proc_features; -/* Set default isolation type */ -uint64_t pt_isolation =MSHV_PT_ISOLATION_NONE ; + ret =get_host_partition_property (mshv_fd, +HV_PARTITION_PROPERTY_PROCESSOR_FEATURES1 , + &host_proc_features); +if (ret <0 ) { +error_report ("Failed to get host processor feature bank 1" ); +return -1 ; + } + args.pt_cpu_fbanks[1 ] = ~host_proc_features; +/* populate args structure */ args.pt_flags = pt_flags; - args.pt_isolation = pt_isolation; + args.pt_isolation =MSHV_PT_ISOLATION_NONE ; + args.pt_disabled_xsave = disabled_xsave_features.as_uint64; + args.pt_num_cpu_fbanks =MSHV_NUM_CPU_FEATURES_BANKS ; ret =ioctl (mshv_fd,MSHV_CREATE_PARTITION , &args);if (ret <0 ) {include/hw/hyperv/hvgdk_mini.h +2
-0@@ -456,6 +456,8 @@ typedef struct hv_input_set_vp_registers {struct hv_register_assoc elements[]; }QEMU_PACKED hv_input_set_vp_registers; +#define MSHV_VP_MAX_REGISTERS 128 +union hv_interrupt_control {uint64_t as_uint64;struct {include/hw/hyperv/hvhdk.h +195
-0@@ -13,6 +13,16 @@#define HV_PARTITION_SYNTHETIC_PROCESSOR_FEATURES_BANKS 1 +struct hv_input_get_partition_property { +uint64_t partition_id; +uint32_t property_code;/* enum hv_partition_property_code */ +uint32_t padding; +}QEMU_PACKED ; + +struct hv_output_get_partition_property { +uint64_t property_value; +}QEMU_PACKED ; +struct hv_input_set_partition_property {uint64_t partition_id;uint32_t property_code;/* enum hv_partition_property_code */ @@ -163,6 +173,191 @@ union hv_partition_synthetic_processor_features { }; }; +union hv_partition_processor_xsave_features { +struct { +uint64_t xsave_support:1 ; +uint64_t xsaveopt_support:1 ; +uint64_t avx_support:1 ; +uint64_t avx2_support:1 ; +uint64_t fma_support:1 ; +uint64_t mpx_support:1 ; +uint64_t avx512_support:1 ; +uint64_t avx512_dq_support:1 ; +uint64_t avx512_cd_support:1 ; +uint64_t avx512_bw_support:1 ; +uint64_t avx512_vl_support:1 ; +uint64_t xsave_comp_support:1 ; +uint64_t xsave_supervisor_support:1 ; +uint64_t xcr1_support:1 ; +uint64_t avx512_bitalg_support:1 ; +uint64_t avx512_i_fma_support:1 ; +uint64_t avx512_v_bmi_support:1 ; +uint64_t avx512_v_bmi2_support:1 ; +uint64_t avx512_vnni_support:1 ; +uint64_t gfni_support:1 ; +uint64_t vaes_support:1 ; +uint64_t avx512_v_popcntdq_support:1 ; +uint64_t vpclmulqdq_support:1 ; +uint64_t avx512_bf16_support:1 ; +uint64_t avx512_vp2_intersect_support:1 ; +uint64_t avx512_fp16_support:1 ; +uint64_t xfd_support:1 ; +uint64_t amx_tile_support:1 ; +uint64_t amx_bf16_support:1 ; +uint64_t amx_int8_support:1 ; +uint64_t avx_vnni_support:1 ; +uint64_t avx_ifma_support:1 ; +uint64_t avx_ne_convert_support:1 ; +uint64_t avx_vnni_int8_support:1 ; +uint64_t avx_vnni_int16_support:1 ; +uint64_t avx10_1_256_support:1 ; +uint64_t avx10_1_512_support:1 ; +uint64_t amx_fp16_support:1 ; +uint64_t reserved1:26 ; + }; +uint64_t as_uint64; +}; + +#define HV_PARTITION_PROCESSOR_FEATURES_BANKS 2 +#define HV_PARTITION_PROCESSOR_FEATURES_RESERVEDBANK1_BITFIELD_COUNT 4 + + +union hv_partition_processor_features { +uint64_t as_uint64[HV_PARTITION_PROCESSOR_FEATURES_BANKS ]; +struct { +uint64_t sse3_support:1 ; +uint64_t lahf_sahf_support:1 ; +uint64_t ssse3_support:1 ; +uint64_t sse4_1_support:1 ; +uint64_t sse4_2_support:1 ; +uint64_t sse4a_support:1 ; +uint64_t xop_support:1 ; +uint64_t pop_cnt_support:1 ; +uint64_t cmpxchg16b_support:1 ; +uint64_t altmovcr8_support:1 ; +uint64_t lzcnt_support:1 ; +uint64_t mis_align_sse_support:1 ; +uint64_t mmx_ext_support:1 ; +uint64_t amd3dnow_support:1 ; +uint64_t extended_amd3dnow_support:1 ; +uint64_t page_1gb_support:1 ; +uint64_t aes_support:1 ; +uint64_t pclmulqdq_support:1 ; +uint64_t pcid_support:1 ; +uint64_t fma4_support:1 ; +uint64_t f16c_support:1 ; +uint64_t rd_rand_support:1 ; +uint64_t rd_wr_fs_gs_support:1 ; +uint64_t smep_support:1 ; +uint64_t enhanced_fast_string_support:1 ; +uint64_t bmi1_support:1 ; +uint64_t bmi2_support:1 ; +uint64_t hle_support_deprecated:1 ; +uint64_t rtm_support_deprecated:1 ; +uint64_t movbe_support:1 ; +uint64_t npiep1_support:1 ; +uint64_t dep_x87_fpu_save_support:1 ; +uint64_t rd_seed_support:1 ; +uint64_t adx_support:1 ; +uint64_t intel_prefetch_support:1 ; +uint64_t smap_support:1 ; +uint64_t hle_support:1 ; +uint64_t rtm_support:1 ; +uint64_t rdtscp_support:1 ; +uint64_t clflushopt_support:1 ; +uint64_t clwb_support:1 ; +uint64_t sha_support:1 ; +uint64_t x87_pointers_saved_support:1 ; +uint64_t invpcid_support:1 ; +uint64_t ibrs_support:1 ; +uint64_t stibp_support:1 ; +uint64_t ibpb_support:1 ; +uint64_t unrestricted_guest_support:1 ; +uint64_t mdd_support:1 ; +uint64_t fast_short_rep_mov_support:1 ; +uint64_t l1dcache_flush_support:1 ; +uint64_t rdcl_no_support:1 ; +uint64_t ibrs_all_support:1 ; +uint64_t skip_l1df_support:1 ; +uint64_t ssb_no_support:1 ; +uint64_t rsb_a_no_support:1 ; +uint64_t virt_spec_ctrl_support:1 ; +uint64_t rd_pid_support:1 ; +uint64_t umip_support:1 ; +uint64_t mbs_no_support:1 ; +uint64_t mb_clear_support:1 ; +uint64_t taa_no_support:1 ; +uint64_t tsx_ctrl_support:1 ; +uint64_t reserved_bank0:1 ; + +/* N.B. Begin bank 1 processor features. */ +uint64_t a_count_m_count_support:1 ; +uint64_t tsc_invariant_support:1 ; +uint64_t cl_zero_support:1 ; +uint64_t rdpru_support:1 ; +uint64_t la57_support:1 ; +uint64_t mbec_support:1 ; +uint64_t nested_virt_support:1 ; +uint64_t psfd_support:1 ; +uint64_t cet_ss_support:1 ; +uint64_t cet_ibt_support:1 ; +uint64_t vmx_exception_inject_support:1 ; +uint64_t enqcmd_support:1 ; +uint64_t umwait_tpause_support:1 ; +uint64_t movdiri_support:1 ; +uint64_t movdir64b_support:1 ; +uint64_t cldemote_support:1 ; +uint64_t serialize_support:1 ; +uint64_t tsc_deadline_tmr_support:1 ; +uint64_t tsc_adjust_support:1 ; +uint64_t fzl_rep_movsb:1 ; +uint64_t fs_rep_stosb:1 ; +uint64_t fs_rep_cmpsb:1 ; +uint64_t tsx_ld_trk_support:1 ; +uint64_t vmx_ins_outs_exit_info_support:1 ; +uint64_t hlat_support:1 ; +uint64_t sbdr_ssdp_no_support:1 ; +uint64_t fbsdp_no_support:1 ; +uint64_t psdp_no_support:1 ; +uint64_t fb_clear_support:1 ; +uint64_t btc_no_support:1 ; +uint64_t ibpb_rsb_flush_support:1 ; +uint64_t stibp_always_on_support:1 ; +uint64_t perf_global_ctrl_support:1 ; +uint64_t npt_execute_only_support:1 ; +uint64_t npt_ad_flags_support:1 ; +uint64_t npt1_gb_page_support:1 ; +uint64_t amd_processor_topology_node_id_support:1 ; +uint64_t local_machine_check_support:1 ; +uint64_t extended_topology_leaf_fp256_amd_support:1 ; +uint64_t gds_no_support:1 ; +uint64_t cmpccxadd_support:1 ; +uint64_t tsc_aux_virtualization_support:1 ; +uint64_t rmp_query_support:1 ; +uint64_t bhi_no_support:1 ; +uint64_t bhi_dis_support:1 ; +uint64_t prefetch_i_support:1 ; +uint64_t sha512_support:1 ; +uint64_t mitigation_ctrl_support:1 ; +uint64_t rfds_no_support:1 ; +uint64_t rfds_clear_support:1 ; +uint64_t sm3_support:1 ; +uint64_t sm4_support:1 ; +uint64_t secure_avic_support:1 ; +uint64_t guest_intercept_ctrl_support:1 ; +uint64_t sbpb_supported:1 ; +uint64_t ibpb_br_type_supported:1 ; +uint64_t srso_no_supported:1 ; +uint64_t srso_user_kernel_no_supported:1 ; +uint64_t vrew_clear_supported:1 ; +uint64_t tsa_l1_no_supported:1 ; +uint64_t tsa_sq_no_supported:1 ; +uint64_t lass_support:1 ; +uint64_t idle_hlt_intercept_support:1 ; +uint64_t msr_list_support:1 ; + }; +}; +enum hv_translate_gva_result_code {HV_TRANSLATE_GVA_SUCCESS =0 ,Signature · 1 line
2.34.1
Re: [PATCH v2 1/9] accel/mshv: use mshv_create_partition_v2 payload
Quoted 3 lines · Anirudh Rayabharam
Why not just call the param
property_codeinstead offeature_bank? That way this function can be used for other properties too.
makes sense
Quoted 5 lines
Quoted 1 line
+
QEMU_BUILD_BUG_ON(ARRAY_SIZE(feature_banks) > MSHV_NUM_CPU_FEATURES_BANKS);Is this really useful since we're not using the
feature_banksarray anywhere?
yeah that's arguable. the intent is to detect drift between the number of feature banks defined in the header and what we handle in the args. we can just assert == 2 for now.
thanks,
magnus
PATCH v2 2/9 target/i386/mshv: fix cpuid propagation bug
In reply to Magnus KulkeThe list argument was missing a pointer and hence e.g. -cpu qemu64 and -cpu EPYC would produce the same of cpuid leaves.
--- target/i386/mshv/mshv-cpu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
target/i386/mshv/mshv-cpu.c +4 -4
@@ -441,7 +441,7 @@ int mshv_load_regs(CPUState *cpu)return 0 ; } -static void add_cpuid_entry (GList *cpuid_entries, +static void add_cpuid_entry (GList **cpuid_entries,uint32_t function,uint32_t index,uint32_t eax,uint32_t ebx,uint32_t ecx,uint32_t edx)
@@ -456,10 +456,10 @@ static void add_cpuid_entry(GList *cpuid_entries, entry->ecx = ecx; entry->edx = edx; - cpuid_entries =g_list_append (cpuid_entries, entry); + *cpuid_entries =g_list_append (*cpuid_entries, entry); } -static void collect_cpuid_entries (const CPUState *cpu,GList *cpuid_entries) +static void collect_cpuid_entries (const CPUState *cpu,GList **cpuid_entries) {X86CPU *x86_cpu =X86_CPU (cpu);CPUX86State *env = &x86_cpu->env;
@@ -615,7 +615,7 @@ static int set_cpuid2(const CPUState *cpu)struct hv_cpuid_entry *entry;GList *entries =NULL ; -collect_cpuid_entries (cpu, entries); +collect_cpuid_entries (cpu, &entries); n_entries =g_list_length (entries); cpuid_size =sizeof (struct hv_cpuid )
Signature · 1 line
2.34.1
Re: [PATCH v2 2/9] target/i386/mshv: fix cpuid propagation bug
Quoted 3 lines · Magnus Kulke
The list argument was missing a pointer and hence e.g. -cpu qemu64 and -cpu EPYC would produce the same of cpuid leaves.
Quoted 4 lines
Signed-off-by Magnus Kulke <[email protected]>--- target/i386/mshv/mshv-cpu.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
PATCH v2 3/9 target/i386/mshv: fix various cpuid traversal bugs
In reply to Magnus Kulke- Hardcoded max_leaf was not accurate. We query leaf 0x0 and 0x80000000 to get the actual max leaves
- On all 0 zeroes on leaf 0x0d, we register 0-63 subleaves with zeros indicating XSAVE is disabled
- Subleaf 0 was hardcoded, so the Hypervisor returned defaults for other subleaves
- Subleaf 0 war hardcoded, so we were passing 0 instead of actual subleaf when adding entries. We now pass the correct subleaf value to add_cpuid_entry()
- Leaves 0x04,0x07,0d,0f,10 weren't marked as subleaf-specific
--- target/i386/mshv/mshv-cpu.c | 88 +++++++++++++++++++++++++------------ 1 file changed, 61 insertions(+), 27 deletions(-)
target/i386/mshv/mshv-cpu.c +61 -27
@@ -465,14 +465,17 @@ static void collect_cpuid_entries(const CPUState *cpu, GList **cpuid_entries)CPUX86State *env = &x86_cpu->env;uint32_t eax, ebx, ecx, edx;uint32_t leaf, subleaf; -size_t max_leaf =0x1F ; -size_t max_subleaf =0x20 ; - -uint32_t leaves_with_subleaves[] = {0x4 ,0x7 ,0xD ,0xF ,0x10 }; +uint32_t max_basic_leaf, max_extended_leaf; +uint32_t max_subleaf =0x20 ; +uint32_t leaves_with_subleaves[] = {0x04 ,0x07 ,0x0d ,0x0f ,0x10 };int n_subleaf_leaves =ARRAY_SIZE (leaves_with_subleaves); -/* Regular leaves without subleaves */ -for (leaf =0 ; leaf <= max_leaf; leaf++) { +/* Get maximum basic and and extended CPUID leaves */ +cpu_x86_cpuid (env,0 ,0 , &max_basic_leaf, &ebx, &ecx, &edx); +cpu_x86_cpuid (env,0x80000000 ,0 , &max_extended_leaf, &ebx, &ecx, &edx); + +/* Collect basic leaves (0x0 to max_basic_leaf) */ +for (leaf =0 ; leaf <= max_basic_leaf; leaf++) {bool has_subleaves = false;for (int i =0 ; i < n_subleaf_leaves; i++) {if (leaf == leaves_with_subleaves[i]) {
@@ -483,27 +486,40 @@ static void collect_cpuid_entries(const CPUState *cpu, GList **cpuid_entries)if (!has_subleaves) {cpu_x86_cpuid (env, leaf,0 , &eax, &ebx, &ecx, &edx); -if (eax ==0 && ebx ==0 && ecx ==0 && edx ==0 ) { -/* all zeroes indicates no more leaves */ -continue ; - } -add_cpuid_entry (cpuid_entries, leaf,0 , eax, ebx, ecx, edx);continue ; } +/* +* Valid XSAVE components can exist at a higher index se we need to set +* all subleaves for leaf 0x0d, even if we encounter an empty one. +*/ +if (leaf ==0x0d ) { +for (subleaf =0 ; subleaf <=63 ; subleaf++) { +cpu_x86_cpuid (env, leaf, subleaf, &eax, &ebx, &ecx, &edx); +add_cpuid_entry (cpuid_entries, leaf, subleaf, + eax, ebx, ecx, edx); + } +continue ; + } + subleaf =0 ;while (subleaf < max_subleaf) {cpu_x86_cpuid (env, leaf, subleaf, &eax, &ebx, &ecx, &edx);if (eax ==0 && ebx ==0 && ecx ==0 && edx ==0 ) { -/* all zeroes indicates no more leaves */ break ; } -add_cpuid_entry (cpuid_entries, leaf,0 , eax, ebx, ecx, edx); +add_cpuid_entry (cpuid_entries, leaf, subleaf, eax, ebx, ecx, edx); subleaf++; } } + +/* Collect extended leaves (0x80000000 to max_extended_leaf) */ +for (leaf =0x80000000 ; leaf <= max_extended_leaf; leaf++) { +cpu_x86_cpuid (env, leaf,0 , &eax, &ebx, &ecx, &edx); +add_cpuid_entry (cpuid_entries, leaf,0 , eax, ebx, ecx, edx); + } } static int register_intercept_result_cpuid_entry(const CPUState *cpu,
@@ -576,22 +592,40 @@ static int register_intercept_result_cpuid(const CPUState *cpu, subleaf_specific =0 ; always_override =1 ; -/* Intel */ -/* 0xb - Extended Topology Enumeration Leaf */ -/* 0x1f - V2 Extended Topology Enumeration Leaf */ -/* AMD */ -/* 0x8000_001e - Processor Topology Information */ -/* 0x8000_0026 - Extended CPU Topology */ -if (entry->function ==0xb - || entry->function ==0x1f - || entry->function ==0x8000001e - || entry->function ==0x80000026 ) { +/* +* Intel +* 0xb - Extended Topology Enumeration Leaf +* 0x1f - V2 Extended Topology Enumeration Leaf +* AMD +* 0x8000_001e - Processor Topology Information +* 0x8000_0026 - Extended CPU Topology +*/ +if (entry->function ==0xb || + entry->function ==0x1f || + entry->function ==0x8000001e || + entry->function ==0x80000026 ) { + subleaf_specific =1 ; + always_override =1 ; +/* +* Feature enumeration leaves (subleaf-specific) +* 0x04: Deterministic Cache Parameters +* 0x07: Structured Extended Feature Flags +* 0x0D: Processor Extended State Enumeration +* 0x0F: Platform QoS Monitoring +* 0x10: Platform QoS Enforcement +*/ + }else if (entry->function ==0x04 || + entry->function ==0x07 || + entry->function ==0x0d || + entry->function ==0x0f || + entry->function ==0x10 ) { subleaf_specific =1 ; always_override =1 ; - }else if (entry->function ==0x00000001 - || entry->function ==0x80000000 - || entry->function ==0x80000001 - || entry->function ==0x80000008 ) { +/* Basic feature leaves (no subleaves) */ + }else if (entry->function ==0x00000001 || + entry->function ==0x80000000 || + entry->function ==0x80000001 || + entry->function ==0x80000008 ) { subleaf_specific =0 ; always_override =1 ; }
Signature · 1 line
2.34.1
Re: [PATCH v2 3/9] target/i386/mshv: fix various cpuid traversal bugs
Quoted 9 lines · Magnus Kulke
- Hardcoded
max_leafwas not accurate. We query leaf0x0and0x80000000to get the actual max leaves- On all 0 zeroes on leaf
0x0d, we register 0-63 subleaves with zeros indicating XSAVE is disabled- Subleaf 0 was hardcoded, so the Hypervisor returned defaults for other subleaves
Quoted 7 lines
- Subleaf 0 war hardcoded, so we were passing 0 instead of actual subleaf when adding entries. We now pass the correct subleaf value to
add_cpuid_entry()- Leaves 0x04,0x07,0d,0f,10 weren't marked as subleaf-specific
Signed-off-by Magnus Kulke <[email protected]>
I didn't check whether the changes matched what the manuals say, but if things were not working before but are working now, then I guess the changes are correct.
PATCH v2 4/9 target/i386/mshv: change cpuid mask to UINT32_MAX
In reply to Magnus KulkeThe current implementation would only override the feature bits that are enabled, however we also want to consider disabled features, hence all bits are set on the masks in the hypercall argument.
--- target/i386/mshv/mshv-cpu.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-)
target/i386/mshv/mshv-cpu.c +6 -11
@@ -538,22 +538,17 @@ static int register_intercept_result_cpuid_entry(const CPUState *cpu, .input.always_override = always_override, .input.padding = 0,/* -* With regard to masks - these are to specify bits to be overwritten -* The current CpuidEntry structure wouldn't allow to carry the masks -* in addition to the actual register values. For this reason, the -* masks are set to the exact values of the corresponding register bits -* to be registered for an overwrite. To view resulting values the -* hypervisor would return, HvCallGetVpCpuidValues hypercall can be -* used. +* Masks specify which bits to override. Set to 0xFFFFFFFF to +* override all bits with the values from the QEMU CPU model. */ .result.eax = entry->eax, - .result.eax_mask = entry->eax, + .result.eax_mask = 0xFFFFFFFF, .result.ebx = entry->ebx, - .result.ebx_mask = entry->ebx, + .result.ebx_mask = 0xFFFFFFFF, .result.ecx = entry->ecx, - .result.ecx_mask = entry->ecx, + .result.ecx_mask = 0xFFFFFFFF, .result.edx = entry->edx, - .result.edx_mask = entry->edx, + .result.edx_mask = 0xFFFFFFFF, };union hv_register_intercept_result_parameters parameters = { .cpuid = cpuid_params,
Signature · 1 line
2.34.1
Re: [PATCH v2 4/9] target/i386/mshv: change cpuid mask to UINT32_MAX
Quoted 5 lines · Magnus Kulke
The current implementation would only override the feature bits that are enabled, however we also want to consider disabled features, hence all bits are set on the masks in the hypercall argument.
Signed-off-by Magnus Kulke <[email protected]>
PATCH v2 5/9 target/i386/mshv: set cpu model name on -cpu host
In reply to Magnus KulkeBy registering a cpu_accel_class for mshv, we can report an accurate model name on -cpu host.
--- target/i386/mshv/mshv-cpu.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
target/i386/mshv/mshv-cpu.c +31 -0
@@ -24,10 +24,13 @@#include "hw/i386/apic_internal.h" #include "cpu.h" +#include "host-cpu.h" #include "emulate/x86_decode.h" #include "emulate/x86_emu.h" #include "emulate/x86_flags.h" +#include "accel/accel-cpu-target.h" +#include "trace-accel_mshv.h" #include "trace.h"
@@ -1675,3 +1678,31 @@ int mshv_arch_post_init_vm(int vm_fd)return ret; } + +static void mshv_cpu_instance_init (CPUState *cs) +{ +X86CPU *cpu =X86_CPU (cs); + +host_cpu_instance_init (cpu); +} + +static void mshv_cpu_accel_class_init (ObjectClass *oc,const void *data) +{ +AccelCPUClass *acc =ACCEL_CPU_CLASS (oc); + + acc->cpu_instance_init = mshv_cpu_instance_init; +} + +static const TypeInfo mshv_cpu_accel_type_info = { + .name =ACCEL_CPU_NAME ("mshv" ), + .parent =TYPE_ACCEL_CPU , + .class_init = mshv_cpu_accel_class_init, + .abstract = true, +}; + +static void mshv_cpu_accel_register_types (void ) +{ +type_register_static (&mshv_cpu_accel_type_info); +} + +type_init (mshv_cpu_accel_register_types);
Signature · 1 line
2.34.1
Re: [PATCH v2 5/9] target/i386/mshv: set cpu model name on -cpu host
Quoted 4 lines · Magnus Kulke
By registering a
cpu_accel_classfor mshv, we can report an accurate model name on -cpu host.Signed-off-by Magnus Kulke <[email protected]>
PATCH v2 6/9 target/i386: query mshv accel for supported cpuids
In reply to Magnus KulkeWe implement mshv_get_supported_cpuid() and invoke it in x86_cpu_get_supported_feature_word() retrieve the cpu features that the host is supporting. Initially we mask the virtualization capabilitities potentially we might need to mask more in the future.
--- include/system/mshv.h | 3 +++ target/i386/cpu.c | 8 ++++++++ target/i386/mshv/mshv-cpu.c | 27 +++++++++++++++++++++++++++ 3 files changed, 38 insertions(+)
include/system/mshv.h +3 -0
@@ -60,4 +60,7 @@ int mshv_irqchip_add_irqfd_notifier_gsi(const EventNotifier *n, const EventNotifier *rn, int virq);int mshv_irqchip_remove_irqfd_notifier_gsi (const EventNotifier *n,int virq); +/* cpuid */ +uint32_t mshv_get_supported_cpuid (uint32_t func,uint32_t idx,int reg); +#endif
target/i386/cpu.c +8 -0
@@ -26,6 +26,7 @@#include "tcg/helper-tcg.h" #include "exec/translation-block.h" #include "system/hvf.h" +#include "system/mshv.h" #include "hvf/hvf-i386.h" #include "kvm/kvm_i386.h" #include "kvm/tdx.h"
@@ -7997,6 +7998,13 @@ uint64_t x86_cpu_get_supported_feature_word(X86CPU *cpu, FeatureWord w) r =hvf_get_supported_cpuid (wi->cpuid.eax, wi->cpuid.ecx, wi->cpuid.reg); + }else if (mshv_enabled ()) { +if (wi->type !=CPUID_FEATURE_WORD ) { +return 0 ; + } + r =mshv_get_supported_cpuid (wi->cpuid.eax, + wi->cpuid.ecx, + wi->cpuid.reg); }else if (tcg_enabled ()) { r = wi->tcg_features; }else {
target/i386/mshv/mshv-cpu.c +27 -0
@@ -1639,6 +1639,33 @@ void mshv_arch_destroy_vcpu(CPUState *cpu)g_clear_pointer (&env->emu_mmio_buf, g_free); } +uint32_t mshv_get_supported_cpuid (uint32_t func,uint32_t idx,int reg) +{ +uint32_t eax, ebx, ecx, edx; +uint32_t ret =0 ; + +host_cpuid (func, idx, &eax, &ebx, &ecx, &edx); +switch (reg) { +case R_EAX : + ret = eax;break ; +case R_EBX : + ret = ebx;break ; +case R_ECX : + ret = ecx;break ; +case R_EDX : + ret = edx;break ; + } + +/* Disable nested virtualization features not yet supported by MSHV */ +if (func ==0x80000001 && reg ==R_ECX ) { + ret &= ~CPUID_EXT3_SVM ; + } +if (func ==0x01 && reg ==R_ECX ) { + ret &= ~CPUID_EXT_VMX ; + } +return ret; +} +/* * Default Microsoft Hypervisor behavior for unimplemented MSR is to send a * fault to the guest if it tries to access it. It is possible to override
Signature · 1 line
2.34.1
PATCH v2 7/9 target/i386/mshv: populate xsave area offsets
In reply to Magnus KulkeThis function is copied verbatim from kvm_cpu_xsave_init(), for MSHV we need to populate the offsets similarly.
--- target/i386/mshv/mshv-cpu.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
target/i386/mshv/mshv-cpu.c +31 -0
@@ -1706,11 +1706,42 @@ int mshv_arch_post_init_vm(int vm_fd)return ret; } +static void mshv_cpu_xsave_init (void ) +{ +static bool first = true; +uint32_t eax, ebx, ecx, edx; +int i; + +if (!first) { +return ; + } + first = false; + +/* x87 and SSE states are in the legacy region of the XSAVE area. */ + x86_ext_save_areas[XSTATE_FP_BIT ].offset =0 ; + x86_ext_save_areas[XSTATE_SSE_BIT ].offset =0 ; + +for (i =XSTATE_SSE_BIT +1 ; i <XSAVE_STATE_AREA_COUNT ; i++) { +ExtSaveArea *esa = &x86_ext_save_areas[i]; + +if (!esa->size) { +continue ; + } +host_cpuid (0xd , i, &eax, &ebx, &ecx, &edx); +if (eax !=0 ) { +assert (esa->size == eax); + esa->offset = ebx; + esa->ecx = ecx; + } + } +} +static void mshv_cpu_instance_init (CPUState *cs) {X86CPU *cpu =X86_CPU (cs);host_cpu_instance_init (cpu); +mshv_cpu_xsave_init (); }static void mshv_cpu_accel_class_init (ObjectClass *oc,const void *data)
Signature · 1 line
2.34.1
PATCH v2 8/9 target/i386/mshv: filter out CET bits in cpuid
In reply to Magnus Kulkee46dcef1d0 introduced CET_U/CET_S xstate bits in the ExtSaveArea. The response in EBX of a 0x4,1 cpuid query will contain a size contingent on what the guest enabled in the IA32_XSS MSR (compacted xsave size).
Since the MSHV accelerator currently responds with static answers to 0x4d cpuid queries, we'll have to disable those bits, even if the host supports them.
--- target/i386/mshv/mshv-cpu.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
target/i386/mshv/mshv-cpu.c +19 -0
@@ -1663,6 +1663,25 @@ uint32_t mshv_get_supported_cpuid(uint32_t func, uint32_t idx, int reg)if (func ==0x01 && reg ==R_ECX ) { ret &= ~CPUID_EXT_VMX ; } + +/* +* MSHV currently uses static CPUID intercept results for leaf 0xD. +* However there are feature-responses that are dynamic based on what a +* guest enables in XCR0 and XSS, such as CET shadow stack. +* +* A guest which doesn't know about those features yet would encounter an +* unexpcted CPUID[0xD,1].EBX (compactes XSAVE size) and either fail +* or gracefully degrade by not using XSAVE at all. +* +* To avoid this, we filter out supervisor xstate features. +*/ +if (func ==0x07 && idx ==0 && reg ==R_ECX ) { + ret &= ~CPUID_7_0_ECX_CET_SHSTK ; + } +if (func ==0x07 && idx ==0 && reg ==R_EDX ) { + ret &= ~CPUID_7_0_EDX_CET_IBT ; + } +return ret; }
Signature · 1 line
2.34.1
PATCH v2 9/9 accel/mshv: disable la57 (5lvl paging)
In reply to Magnus KulkeThis change disable la57 paging on the mshv hypervisor on both the mshv processor feature bitmap and mask the cpuid feature leaf to the guest.
Since the removal of hypervisor-assisted gva=>gpa translation in 1c85a4a3d7 we have seen MMIO errors in guests on la57-enabled hw. We will have to investigate and test this further.
--- accel/mshv/mshv-all.c | 7 +++++++ target/i386/mshv/mshv-cpu.c | 6 ++++++ 2 files changed, 13 insertions(+)
accel/mshv/mshv-all.c +7 -0
@@ -142,6 +142,8 @@ static int create_partition(int mshv_fd, int *vm_fd)int ret;uint64_t pt_flags, host_proc_features;union hv_partition_processor_xsave_features disabled_xsave_features; +union hv_partition_processor_features disabled_proc_features = {0 }; +struct mshv_create_partition_v2 args = {0 };uint32_t feature_banks[] = {HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 ,
@@ -181,6 +183,11 @@ static int create_partition(int mshv_fd, int *vm_fd) } args.pt_cpu_fbanks[1 ] = ~host_proc_features; +/* features we disable regardless of host support */ + disabled_proc_features.la57_support =1 ; + args.pt_cpu_fbanks[0 ] |= disabled_proc_features.as_uint64[0 ]; + args.pt_cpu_fbanks[1 ] |= disabled_proc_features.as_uint64[1 ]; +/* populate args structure */ args.pt_flags = pt_flags; args.pt_isolation =MSHV_PT_ISOLATION_NONE ;
target/i386/mshv/mshv-cpu.c +6 -0
@@ -1677,6 +1677,12 @@ uint32_t mshv_get_supported_cpuid(uint32_t func, uint32_t idx, int reg)*/ if (func ==0x07 && idx ==0 && reg ==R_ECX ) { ret &= ~CPUID_7_0_ECX_CET_SHSTK ; +/* +* LA57 (5-level paging) causes incorrect GVA=>GPA translations +* in the instruction decoder/emulator. Disable until page table +* walk in x86_mmu.c works w/ 5-level paging. +*/ + ret &= ~CPUID_7_0_ECX_LA57 ; }if (func ==0x07 && idx ==0 && reg ==R_EDX ) { ret &= ~CPUID_7_0_EDX_CET_IBT ;
Signature · 1 line
2.34.1
Re: [PATCH v2 9/9] accel/mshv: disable la57 (5lvl paging)
Quoted 11 lines · Magnus Kulke
This change disable la57 paging on the mshv hypervisor on both the mshv processor feature bitmap and mask the cpuid feature leaf to the guest.
Since the removal of hypervisor-assisted gva=>gpa translation in
1c85a4a3d7we have seen MMIO errors in guests on la57-enabled hw. We will have to investigate and test this further.
Hello,
This should already be fixed on master since yesterday. Can you repro this?
Thank you,
-Mohamed
Quoted 53 lines
Signed-off-by Magnus Kulke <[email protected]>---
accel/mshv/mshv-all.c| 7 +++++++target/i386/mshv/mshv-cpu.c| 6 ++++++2 files changed, 13 insertions(+)accel/mshv/mshv-all.c +7
-0@@ -142,6 +142,8 @@ static int create_partition(int mshv_fd, int *vm_fd)int ret;uint64_t pt_flags, host_proc_features;union hv_partition_processor_xsave_features disabled_xsave_features; +union hv_partition_processor_features disabled_proc_features = {0 }; +struct mshv_create_partition_v2 args = {0 };uint32_t feature_banks[] = {HV_PARTITION_PROPERTY_PROCESSOR_FEATURES0 ,@@ -181,6 +183,11 @@ static int create_partition(int mshv_fd, int *vm_fd) } args.pt_cpu_fbanks[1 ] = ~host_proc_features; +/* features we disable regardless of host support */ + disabled_proc_features.la57_support =1 ; + args.pt_cpu_fbanks[0 ] |= disabled_proc_features.as_uint64[0 ]; + args.pt_cpu_fbanks[1 ] |= disabled_proc_features.as_uint64[1 ]; +/* populate args structure */ args.pt_flags = pt_flags; args.pt_isolation =MSHV_PT_ISOLATION_NONE ;target/i386/mshv/mshv-cpu.c +6
-0@@ -1677,6 +1677,12 @@ uint32_t mshv_get_supported_cpuid(uint32_t func, uint32_t idx, int reg)*/ if (func ==0x07 && idx ==0 && reg ==R_ECX ) { ret &= ~CPUID_7_0_ECX_CET_SHSTK ; +/* +* LA57 (5-level paging) causes incorrect GVA=>GPA translations +* in the instruction decoder/emulator. Disable until page table +* walk in x86_mmu.c works w/ 5-level paging. +*/ + ret &= ~CPUID_7_0_ECX_LA57 ; }if (func ==0x07 && idx ==0 && reg ==R_EDX ) { ret &= ~CPUID_7_0_EDX_CET_IBT ;Signature · 1 line
2.34.1
Re: [PATCH v2 9/9] accel/mshv: disable la57 (5lvl paging)
Quoted 2 lines · Mohamed Mediouni
This should already be fixed on master since yesterday. Can you repro this?
Hey Mohamed,
if you refer to "cb02fc2284 target/i386: emulate: LA57 fix", I don't think that was sufficient. It addressed a stack smashing issue, but the guest would still run into MMIO problems on la57-enabled hw.
Re: [PATCH v2 9/9] accel/mshv: disable la57 (5lvl paging)
Quoted 14 lines · Magnus Kulke
Quoted 2 lines · Mohamed Mediouni
This should already be fixed on master since yesterday. Can you repro this?
Hey Mohamed,
if you refer to "
cb02fc2284target/i386: emulate: LA57 fix", I don't think that was sufficient. It addressed a stack smashing issue, but the guest would still run into MMIO problems on la57-enabled hw.
Hello,
Thank you, I’ll test this further. Boot-tested it on an Alpine Linux VM but looks like that wasn’t enough...
FYI I added a new callback interface in x86_emul_ops (mmu_gva_to_gpa) to support adding the Hyper-V page table walker, but if it’s anything like the WHP one it’s going to be very slow… which is why I moved to the new interface in the first place.
Re: [PATCH v2 9/9] accel/mshv: disable la57 (5lvl paging)
Quoted 6 lines · Mohamed Mediouni
Thank you, I’ll test this further. Boot-tested it on an Alpine Linux VM but looks like that wasn’t enough...
FYI I added a new callback interface in
x86_emul_ops(mmu_gva_to_gpa) to support adding the Hyper-V page table walker, but if it’s anything like the WHP one it’s going to be very slow… which is why I moved to the new interface in the first place.
hmm, I quickly tried that and implemented a hook like this:
static MMUTranslateResult gva_to_gpa(CPUState *cpu, target_ulong gva,
uint64_t *gpa, MMUTranslateFlags flags)
{
uint64_t hv_flags = 0;
if (!x86_is_paging_mode(cpu)) {
*gpa = gva;
return MMU_TRANSLATE_SUCCESS;
}
if (flags & MMU_TRANSLATE_VALIDATE_WRITE) {
hv_flags = HV_TRANSLATE_GVA_VALIDATE_WRITE;
} else if (flags & MMU_TRANSLATE_VALIDATE_EXECUTE) {
hv_flags = HV_TRANSLATE_GVA_VALIDATE_EXECUTE;
} else {
hv_flags = HV_TRANSLATE_GVA_VALIDATE_READ;
}
if (translate_gva(cpu, gva, gpa, hv_flags) < 0) {
return MMU_TRANSLATE_PAGE_NOT_MAPPED;
}
return MMU_TRANSLATE_SUCCESS;
}static const struct x86_emul_ops mshv_x86_emul_ops = {
.read_segment_descriptor = read_segment_descriptor, .mmu_gva_to_gpa = gva_to_gpa,
};
but without explicitly disabling la57, I still see the same MMIO errors during guest boot (ubuntu jammy). probably needs more investigation. I think for now disabling la57 seems ok.