KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
PATCH 1/1 KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.
Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when it was set for L1. TSC_MULTIPLIER is already correctly computed and written by prepare_vmcs02().
--- arch/x86/kvm/vmx/nested.c | 1 - 1 file changed, 1 deletion(-)
arch/x86/kvm/vmx/nested.c +0 -1
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |SECONDARY_EXEC_APIC_REGISTER_VIRT |SECONDARY_EXEC_ENABLE_VMFUNC | -SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_DESC); if (nested_cpu_has(vmcs12,
Signature · 1 line
2.35.3
Quoted 8 lines · Vitaly Kuznetsov
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters
SECONDARY_EXEC_TSC_SCALINGout when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.
Ouch.
I guess that needs a Fixes tag?
Also this is thankfully Intel specific, because in AMD you can't enable TSC scaling - there is just an MSR with default value of 1.0, which one can change if TSC scaling is supported in CPUID.
Best regards,
Maxim Levitsky
Quoted 22 lines
Keep
SECONDARY_EXEC_TSC_SCALINGin secondary exec controls for L2 when it was set for L1.TSC_MULTIPLIERis already correctly computed and written byprepare_vmcs02().Signed-off-by Vitaly Kuznetsov <[email protected]>---
arch/x86/kvm/vmx/nested.c | 1 -1 file changed, 1 deletion(-)arch/x86/kvm/vmx/nested.c +0
-0@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY|SECONDARY_EXEC_APIC_REGISTER_VIRT|SECONDARY_EXEC_ENABLE_VMFUNC| -SECONDARY_EXEC_TSC_SCALING|SECONDARY_EXEC_DESC);
if (nested_cpu_has(vmcs12,
Quoted 6 lines · Maxim Levitsky
Quoted 5 lines · Vitaly Kuznetsov
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency
^^^ scaling ^^^
Quoted 11 lines
Quoted 4 lines
for its guests even when the feature is advertised and KVM filters
SECONDARY_EXEC_TSC_SCALINGout when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.Ouch.
I guess that needs a Fixes tag?
Fixes d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")
I dismissed that because prior to d041b5ea93352b SECONDARY_EXEC_TSC_SCALING was filtered out in nested_vmx_setup_ctls_msrs() but now I think I was wrong, SECONDARY_EXEC_TSC_SCALING was likely kept in VMCS02 regardless of that. Will add in v2.
Quoted 5 lines
Also this is thankfully Intel specific, because in AMD you can't enable TSC scaling - there is just an MSR with default value of 1.0, which one can change if TSC scaling is supported in CPUID.
Reviewed-by Maxim Levitsky <[email protected]>
Thanks!
Quoted 28 lines
Best regards,
Maxim LevitskyQuoted 22 lines
Keep
SECONDARY_EXEC_TSC_SCALINGin secondary exec controls for L2 when it was set for L1.TSC_MULTIPLIERis already correctly computed and written byprepare_vmcs02().Signed-off-by Vitaly Kuznetsov <[email protected]>---
arch/x86/kvm/vmx/nested.c | 1 -1 file changed, 1 deletion(-)arch/x86/kvm/vmx/nested.c +0
-0@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY|SECONDARY_EXEC_APIC_REGISTER_VIRT|SECONDARY_EXEC_ENABLE_VMFUNC| -SECONDARY_EXEC_TSC_SCALING|SECONDARY_EXEC_DESC);
if (nested_cpu_has(vmcs12,
Signature · 1 line
Vitaly
Sean Christopherson
maintainerQuoted 20 lines · Vitaly Kuznetsov
Quoted 6 lines · Maxim Levitsky
Quoted 5 lines · Vitaly Kuznetsov
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency
^^^ scaling ^^^
Quoted 9 lines
Quoted 4 lines
for its guests even when the feature is advertised and KVM filters
SECONDARY_EXEC_TSC_SCALINGout when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.Ouch.
I guess that needs a Fixes tag?
Quoted 7 lines
Quoted 2 lines
Fixes d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")I dismissed that because prior to
d041b5ea93352bSECONDARY_EXEC_TSC_SCALINGwas filtered out innested_vmx_setup_ctls_msrs()but now I think I was wrong,SECONDARY_EXEC_TSC_SCALINGwas likely kept in VMCS02 regardless of that. Will add in v2.
Yep, it would have been kept in vmcs02 even though the feature wasn't advertised to the L1 VMM. A Cc for stable is warranted as well.
I added this (with the tags and s/frequency/scaling) to the queue of patches for 5.20 I have lined up for Paolo to consume on his return. Paolo and I haven't hashed out how we'll actually manage anything, i.e. my list is speculative, but unless you hear otherwise, no need to send a v2.
Hi Vitaly,
Quoted 3 lines · Vitaly Kuznetsov
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is
Would you mind helping clarify if it is L1 or L2 that hangs?
The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.
Thank you very much!
Dongli Zhang
Quoted 28 lines
supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters
SECONDARY_EXEC_TSC_SCALINGout when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.Keep
SECONDARY_EXEC_TSC_SCALINGin secondary exec controls for L2 when it was set for L1.TSC_MULTIPLIERis already correctly computed and written byprepare_vmcs02().Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/kvm/vmx/nested.c | 1 - 1 file changed, 1 deletion(-)arch/x86/kvm/vmx/nested.c +0
-1@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |SECONDARY_EXEC_APIC_REGISTER_VIRT |SECONDARY_EXEC_ENABLE_VMFUNC | -SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_DESC); if (nested_cpu_has(vmcs12,
Quoted 12 lines · Dongli Zhang
Hi Vitaly,
Quoted 3 lines · Vitaly Kuznetsov
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is
Would you mind helping clarify if it is L1 or L2 that hangs?
The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.
I think it's L2 but I'm not sure: there's no easy way to interract with L1 (Hyper-V) directly, all the interfaces (UI, network,..) are handled by L2 (Windows). Prior to the observed 'hang' Hyper-V (L1) programms synthetic timer in KVM too far in the future but my guess is that it's doing that on Windows' (L2) behalf, basically just relaying the request. The issue only shows up with 'hv-reenlightenment' + 'hv-frequencies' (in QEMU's terms) features as in this case both Hyper-V (L1) and Windows (L2) trust the information about TSC frequency from KVM but the information turns out to be incorrect for Windows (L2).
Signature · 1 line
Vitaly