whpx: i386: x2apic emulation for kernel-irqchip=off, feature probing
v2 cover whpx: i386: x2apic emulation for kernel-irqchip=off, feature probing
"whpx: i386: disable TbFlushHypercalls for emulated LAPIC" is a bugfix, and "target/i386: emulate: include name of unhandled instruction" is a debugging aid.
"whpx: i386: x2apic emulation" makes things slightly better for Windows 10 users. But I strongly recommend *not* relying on it when possible and using kernel-irqchip=on instead. On Windows 10 however that's more murky because PIC interrupt injection is broken (interrupts don't wake the vCPU from HLT) in that case.
"whpx: i386: wire up feature probing" is yet another commit adding a code path not used on Windows 10. It'll tell the user today which CPU features they set are incompatible with the hardware but it does not sync that to the CPUID view that the guest has.
And then another commit to enable x2apic emulation by default even for kernel-irqchip=off + re-introducing provided by QEMU enlightenments in a more functional form to signal that the x2apic can be used. I'm not aware of the actual vmware freq leaf being used though.
"whpx: x2apic emulation for kernel-irqchip=off follow-up" is rolled into this series.
Changes in v2:
- GCC warned when a variable name was re-used within a different (but overlapping) scope in the same function. It also warned with a -Werror=maybe-uninitialized for the MSR write case. Address those
- make the in-KVM enlightenments path available on Windows 11 too when -M hyperv=off.
Mohamed Mediouni (6):
target/i386: emulate: include name of unhandled instruction
whpx: i386: x2apic emulation
whpx: i386: wire up feature probing
whpx: i386: disable TbFlushHypercalls for emulated LAPIC
whpx: i386: enable x2apic by default for user-mode LAPIC
whpx: i386: reintroduce enlightenments for Windows 10
accel/whpx/whpx-common.c | 2 + include/system/whpx-internal.h | 4 + target/arm/whpx/whpx-all.c | 1 + target/i386/cpu.c | 25 ++++ target/i386/emulate/x86_emu.c | 4 +- target/i386/whpx/whpx-all.c | 265 ++++++++++++++++++++++++++++++++- target/i386/whpx/whpx-i386.h | 4 + 7 files changed, 298 insertions(+), 7 deletions(-) create mode 100644 target/i386/whpx/whpx-i386.h
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 1/6 target/i386: emulate: include name of unhandled instruction
Instead of just the command number, include the instruction name to make debugging easier.
--- target/i386/emulate/x86_emu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
target/i386/emulate/x86_emu.c +2 -2
@@ -1399,8 +1399,8 @@ static void init_cmd_handler(void)bool exec_instruction (CPUX86State *env,struct x86_decode *ins) {if (!_cmd_handler[ins->cmd].handler) { -printf ("Unimplemented handler (" TARGET_FMT_lx") for %d (%x %x)\n" , - env->eip, +printf ("Unimplemented handler (" TARGET_FMT_lx") for %s - %d (%x %x)\n" , + env->eip,decode_cmd_to_string (ins->cmd), ins->cmd, ins->opcode[0 ], ins->opcode_len >1 ? ins->opcode[1 ] :0 ); env->eip += ins->len;
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 2/6 whpx: i386: x2apic emulation
In reply to Mohamed MediouniAdd x2apic emulation to WHPX for the kernel-irqchip=off case.
Unfortunately, it looks like there isn't a workaround available for proper behavior of PIC interrupts when kernel-irqchip=on for Windows 10. The OS is out of support outside of extended security updates so this will not be addressed.
On Windows 11, x2apic will be enabled straight away because Linux enables it even without an IOMMU when using Hyper-V.
For Windows 10, you need: -device intel-iommu,intremap=on,eim=on
The performance impact is worthwhile for multicore guests.
--- target/i386/whpx/whpx-all.c | 134 +++++++++++++++++++++++++++++++++++- 1 file changed, 133 insertions(+), 1 deletion(-)
target/i386/whpx/whpx-all.c +133 -1
@@ -1082,6 +1082,8 @@ HRESULT whpx_set_exception_exit_bitmap(UINT64 exceptions)/* Register for MSR and CPUID exits */ memset (&prop,0 ,sizeof (WHV_PARTITION_PROPERTY )); prop.ExtendedVmExits.X64MsrExit =1 ; + prop.ExtendedVmExits.X64CpuidExit =1 ; +if (exceptions !=0 ) { prop.ExtendedVmExits.ExceptionExit =1 ; }
@@ -1898,6 +1900,18 @@ int whpx_vcpu_run(CPUState *cpu)WHV_REGISTER_NAME reg_names[3 ];UINT32 reg_count;bool is_known_msr =0 ; +uint64_t val; + +if (vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite) { + val = ((uint32_t )vcpu->exit_ctx.MsrAccess.Rax) | + ((uint64_t )(vcpu->exit_ctx.MsrAccess.Rdx) <<32 ); + }else { +/* +* Workaround for [-Werror=maybe-uninitialized] +* with GCC. Not needed with Clang. +*/ + val =0 ; + } reg_names[0 ] = WHvX64RegisterRip; reg_names[1 ] = WHvX64RegisterRax;
@@ -1911,7 +1925,47 @@ int whpx_vcpu_run(CPUState *cpu) && !vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite && !whpx_irqchip_in_kernel()) { is_known_msr =1 ; - reg_values[1 ].Reg32 = (uint32_t )X86_CPU (cpu)->env.apic_bus_freq; + val =X86_CPU (cpu)->env.apic_bus_freq; + } + +if (!whpx_irqchip_in_kernel () && + vcpu->exit_ctx.MsrAccess.MsrNumber ==MSR_IA32_APICBASE ) { + is_known_msr =1 ; +if (!vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite) { +/* Read path unreachable on Hyper-V */ +abort (); + }else { +WHV_REGISTER_VALUE reg = {.Reg64 = val}; +int msr_ret =cpu_set_apic_base (X86_CPU (cpu)->apic_state, val); +if (msr_ret <0 ) { +x86_emul_raise_exception (&X86_CPU (cpu)->env,EXCP0D_GPF ,0 ); + } +whpx_set_reg (cpu, WHvX64RegisterApicBase, reg); + } + } + +if (!whpx_irqchip_in_kernel () && + vcpu->exit_ctx.MsrAccess.MsrNumber >=MSR_APIC_START && + vcpu->exit_ctx.MsrAccess.MsrNumber <=MSR_APIC_END ) { +int index = vcpu->exit_ctx.MsrAccess.MsrNumber -MSR_APIC_START ; +int msr_ret; + is_known_msr =1 ; +if (!vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite) { +bql_lock (); + msr_ret =apic_msr_read (X86_CPU (cpu)->apic_state, index, &val); +bql_unlock (); + reg_values[1 ].Reg64 = val; +if (msr_ret <0 ) { +x86_emul_raise_exception (&X86_CPU (cpu)->env,EXCP0D_GPF ,0 ); + } + }else { +bql_lock (); + msr_ret =apic_msr_write (X86_CPU (cpu)->apic_state, index, val); +bql_unlock (); +if (msr_ret <0 ) { +x86_emul_raise_exception (&X86_CPU (cpu)->env,EXCP0D_GPF ,0 ); + } + } }/* * For all unsupported MSR access we:
@@ -1921,6 +1975,11 @@ int whpx_vcpu_run(CPUState *cpu) reg_count = vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite ?1 :3 ; +if (!vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite) { + reg_values[1 ].Reg32 = (uint32_t )val; + reg_values[2 ].Reg32 = (uint32_t )(val >>32 ); + } +if (!is_known_msr) {trace_whpx_unsupported_msr_access (vcpu->exit_ctx.MsrAccess.MsrNumber, vcpu->exit_ctx.MsrAccess.AccessInfo.IsWrite);
@@ -1939,6 +1998,47 @@ int whpx_vcpu_run(CPUState *cpu) ret =0 ;break ; } +case WHvRunVpExitReasonX64Cpuid: { +WHV_REGISTER_VALUE reg_values[5 ] = {0 }; +WHV_REGISTER_NAME reg_names[5 ]; +UINT32 reg_count =5 ; +X86CPU *x86_cpu =X86_CPU (cpu); +CPUX86State *env = &x86_cpu->env; + + reg_names[0 ] = WHvX64RegisterRip; + reg_names[1 ] = WHvX64RegisterRax; + reg_names[2 ] = WHvX64RegisterRcx; + reg_names[3 ] = WHvX64RegisterRdx; + reg_names[4 ] = WHvX64RegisterRbx; + + reg_values[0 ].Reg64 = + vcpu->exit_ctx.VpContext.Rip + + vcpu->exit_ctx.VpContext.InstructionLength; + + reg_values[1 ].Reg64 = vcpu->exit_ctx.CpuidAccess.DefaultResultRax; + reg_values[2 ].Reg64 = vcpu->exit_ctx.CpuidAccess.DefaultResultRcx; + reg_values[3 ].Reg64 = vcpu->exit_ctx.CpuidAccess.DefaultResultRdx; + reg_values[4 ].Reg64 = vcpu->exit_ctx.CpuidAccess.DefaultResultRbx; + +if (vcpu->exit_ctx.CpuidAccess.Rax ==1 ) { +if (cpu_has_x2apic_feature (env)) { + reg_values[2 ].Reg64 |=CPUID_EXT_X2APIC ; + } + } + + hr = whp_dispatch.WHvSetVirtualProcessorRegisters ( + whpx->partition, + cpu->cpu_index, + reg_names, reg_count, + reg_values); + +if (FAILED (hr)) { +error_report ("WHPX: Failed to set CpuidAccess state " +" registers, hr=%08lx" , hr); + } + ret =0 ; +break ; + }case WHvRunVpExitReasonException:whpx_get_registers (cpu,WHPX_LEVEL_FULL_STATE );
@@ -2136,6 +2236,7 @@ int whpx_accel_init(AccelState *as, MachineState *ms)WHV_PROCESSOR_FEATURES_BANKS processor_features;WHV_PROCESSOR_PERFMON_FEATURES perfmon_features;bool is_legacy_os = false; +UINT32 cpuidExitList[] = {1 }; whpx = &whpx_global;
@@ -2354,6 +2455,7 @@ int whpx_accel_init(AccelState *as, MachineState *ms)/* Register for MSR and CPUID exits */ memset (&prop,0 ,sizeof (WHV_PARTITION_PROPERTY )); prop.ExtendedVmExits.X64MsrExit =1 ; + prop.ExtendedVmExits.X64CpuidExit =1 ; hr = whp_dispatch.WHvSetPartitionProperty( whpx->partition,
@@ -2366,6 +2468,36 @@ int whpx_accel_init(AccelState *as, MachineState *ms) goto error; } +memset (&prop,0 ,sizeof (WHV_PARTITION_PROPERTY )); + prop.X64MsrExitBitmap.UnhandledMsrs =1 ; +if (!whpx_irqchip_in_kernel ()) { + prop.X64MsrExitBitmap.ApicBaseMsrWrite =1 ; + } + + hr = whp_dispatch.WHvSetPartitionProperty ( + whpx->partition, + WHvPartitionPropertyCodeX64MsrExitBitmap, + &prop, +sizeof (WHV_PARTITION_PROPERTY )); +if (FAILED (hr)) { +error_report ("WHPX: Failed to set MSR exit bitmap, hr=%08lx" , hr); + ret = -EINVAL ; + goto error; + } + + hr = whp_dispatch.WHvSetPartitionProperty ( + whpx->partition, + WHvPartitionPropertyCodeCpuidExitList, + cpuidExitList, +RTL_NUMBER_OF (cpuidExitList) *sizeof (UINT32 )); + +if (FAILED (hr)) { +error_report ("WHPX: Failed to set partition CpuidExitList hr=%08lx" , + hr); + ret = -EINVAL ; + goto error; + } +/* * We do not want to intercept any exceptions from the guest, * until we actually start debugging with gdb.
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 3/6 whpx: i386: wire up feature probing
In reply to Mohamed MediouniWindows 10 doesn't have the API for this, so using this only for Windows 11.
--- include/system/whpx-internal.h | 3 ++ target/i386/cpu.c | 25 ++++++++++++++ target/i386/whpx/whpx-all.c | 63 +++++++++++++++++++++++++++++++++- target/i386/whpx/whpx-i386.h | 4 +++ 4 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 target/i386/whpx/whpx-i386.h
include/system/whpx-internal.h +3 -0
@@ -89,6 +89,9 @@ void whpx_apic_get(APICCommonState *s); UINT32 StateSize)) \ X(HRESULT, WHvResetPartition, \ (WHV_PARTITION_HANDLE Partition)) \ + X(HRESULT, WHvGetVirtualProcessorCpuidOutput, \ + (WHV_PARTITION_HANDLE Partition, UINT32 VpIndex, UINT32 Eax, \ + UINT32 Ecx, WHV_CPUID_OUTPUT* CpuidOutput)) #define WHP_DEFINE_TYPE(return_type, function_name, signature) \ typedef return_type (WINAPI *function_name ## _t) signature;
target/i386/cpu.c +25 -0
@@ -26,6 +26,8 @@#include "tcg/helper-tcg.h" #include "exec/translation-block.h" #include "system/hvf.h" +#include "system/whpx.h" +#include "whpx/whpx-i386.h" #include "hvf/hvf-i386.h" #include "kvm/kvm_i386.h" #include "kvm/tdx.h"
@@ -8087,6 +8089,17 @@ uint64_t x86_cpu_get_supported_feature_word(X86CPU *cpu, FeatureWord w) r =hvf_get_supported_cpuid (wi->cpuid.eax, wi->cpuid.ecx, wi->cpuid.reg); + }else if (whpx_enabled ()) { +if (wi->type !=CPUID_FEATURE_WORD ) { +return 0 ; + } +if (whpx_is_legacy_os ()) { + r = wi->tcg_features; + }else { + r =whpx_get_supported_cpuid (wi->cpuid.eax, + wi->cpuid.ecx, + wi->cpuid.reg); + } }else if (tcg_enabled () ||qtest_enabled ()) { r = wi->tcg_features; }else {
@@ -8168,6 +8181,18 @@ static void x86_cpu_get_supported_cpuid(uint32_t func, uint32_t index, *ebx =hvf_get_supported_cpuid (func, index,R_EBX ); *ecx =hvf_get_supported_cpuid (func, index,R_ECX ); *edx =hvf_get_supported_cpuid (func, index,R_EDX ); + }else if (whpx_enabled ()) { +if (whpx_is_legacy_os ()) { + *eax =0 ; + *ebx =0 ; + *ecx =0 ; + *edx =0 ; + }else { + *eax =whpx_get_supported_cpuid (func, index,R_EAX ); + *ebx =whpx_get_supported_cpuid (func, index,R_EBX ); + *ecx =whpx_get_supported_cpuid (func, index,R_ECX ); + *edx =whpx_get_supported_cpuid (func, index,R_EDX ); + } }else { *eax =0 ; *ebx =0 ;
target/i386/whpx/whpx-all.c +62 -1
87 lines not on this page · raw message
target/i386/whpx/whpx-i386.h new file +4 -0
@@ -0,0 +1,4 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +uint32_t whpx_get_supported_cpuid (uint32_t func,uint32_t idx,int reg); +bool whpx_is_legacy_os (void );
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 4/6 whpx: i386: disable TbFlushHypercalls for emulated LAPIC
In reply to Mohamed MediouniAccessHypercallRegs was present twice so clean that up.
Remove TbFlushHypercalls (and its extended Gva range sub-feature) from the user-mode LAPIC case as it behaves oddly there.
--- target/i386/whpx/whpx-all.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
target/i386/whpx/whpx-all.c +6 -3
@@ -2483,10 +2483,7 @@ int whpx_accel_init(AccelState *as, MachineState *ms) synthetic_features.Bank0.AccessPartitionReferenceTsc =1 ; synthetic_features.Bank0.AccessHypercallRegs =1 ; synthetic_features.Bank0.AccessFrequencyRegs =1 ; - synthetic_features.Bank0.EnableExtendedGvaRangesForFlushVirtualAddressList =1 ; synthetic_features.Bank0.AccessVpIndex =1 ; - synthetic_features.Bank0.AccessHypercallRegs =1 ; - synthetic_features.Bank0.TbFlushHypercalls =1 ;if (whpx_irqchip_in_kernel ()) { synthetic_features.Bank0.AccessSynicRegs =1 ;
@@ -2494,6 +2491,12 @@ int whpx_accel_init(AccelState *as, MachineState *ms) synthetic_features.Bank0.AccessIntrCtrlRegs =1 ; synthetic_features.Bank0.SyntheticClusterIpi =1 ; synthetic_features.Bank0.DirectSyntheticTimers =1 ; +/* +* These technically work without the Hyper-V LAPIC +* but behave oddly for multi-core VMs. +*/ + synthetic_features.Bank0.TbFlushHypercalls =1 ; + synthetic_features.Bank0.EnableExtendedGvaRangesForFlushVirtualAddressList =1 ; }if (!is_legacy_os && whpx->hyperv_enlightenments_allowed) {
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 5/6 whpx: i386: enable x2apic by default for user-mode LAPIC
In reply to Mohamed Mediouni--- target/i386/whpx/whpx-all.c | 7 +++++++ 1 file changed, 7 insertions(+)
target/i386/whpx/whpx-all.c +7 -0
@@ -2275,11 +2275,18 @@ error:return ret; } +static PropValue whpx_default_props[] = { + {"x2apic" ,"on" }, + {NULL ,NULL }, +}; + +void whpx_cpu_instance_init (CPUState *cs) {X86CPU *cpu =X86_CPU (cs);host_cpu_instance_init (cpu); +x86_cpu_apply_props (cpu, whpx_default_props); }/*
Signature · 1 line
2.50.1 (Apple Git-155)
PATCH v2 6/6 whpx: i386: reintroduce enlightenments for Windows 10
In reply to Mohamed MediouniWas removed in 2c08624 but it's still useful for Windows 10 so reintroduce it there.
And this time, actually make it work by reporting the hypervisor bit in CPUID.
Also pretend to be KVM, with the only capability reported being X2APIC support.
Although looks like Linux doesn't use that vmware leaf...
--- accel/whpx/whpx-common.c | 2 + include/system/whpx-internal.h | 1 + target/arm/whpx/whpx-all.c | 1 + target/i386/whpx/whpx-all.c | 68 ++++++++++++++++++++++++++++++---- 4 files changed, 64 insertions(+), 8 deletions(-)
accel/whpx/whpx-common.c +2 -0
@@ -550,6 +550,8 @@ static void whpx_accel_instance_init(Object *obj) whpx->hyperv_enlightenments_allowed = true; whpx->hyperv_enlightenments_required = false; +/* Value determined at whpx_accel_init */ + whpx->hyperv_enlightenments_enabled = false; }static const TypeInfo whpx_accel_type = {
include/system/whpx-internal.h +1 -0
@@ -45,6 +45,7 @@ struct whpx_state {bool hyperv_enlightenments_allowed;bool hyperv_enlightenments_required; +bool hyperv_enlightenments_enabled; };
target/arm/whpx/whpx-all.c +1 -0
@@ -968,6 +968,7 @@ int whpx_accel_init(AccelState *as, MachineState *ms)* as they're not needed for performance. */ if (whpx->hyperv_enlightenments_required) { + whpx->hyperv_enlightenments_enabled = true; hr = whp_dispatch.WHvSetPartitionProperty( whpx->partition, WHvPartitionPropertyCodeSyntheticProcessorFeaturesBanks,
target/i386/whpx/whpx-all.c +60 -8
86 lines not on this page · raw message
Signature · 1 line
2.50.1 (Apple Git-155)