Skip to contentSkip to conversation

KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1

PATCH In review Vitaly Kuznetsov 4 participants 6 messages Jul 12, 2022 – Jul 13, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.

Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when it was set for L1. TSC_MULTIPLIER is already correctly computed and written by prepare_vmcs02().

---
 arch/x86/kvm/vmx/nested.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kvm/vmx/nested.c +0 -1
index 778f82015f03..bfa366938c49 100644
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
 				  SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
 				  SECONDARY_EXEC_APIC_REGISTER_VIRT |
 				  SECONDARY_EXEC_ENABLE_VMFUNC |
-				  SECONDARY_EXEC_TSC_SCALING |
 				  SECONDARY_EXEC_DESC);
 
 		if (nested_cpu_has(vmcs12,
Signature · 1 line
2.35.3
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Vitaly Kuznetsov

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.

Ouch.

I guess that needs a Fixes tag?

Fixes d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")

Also this is thankfully Intel specific, because in AMD you can't enable TSC scaling - there is just an MSR with default value of 1.0, which one can change if TSC scaling is supported in CPUID.

Best regards,
Maxim Levitsky

Quoted 22 lines

Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when it was set for L1. TSC_MULTIPLIER is already correctly computed and written by prepare_vmcs02().

---
 arch/x86/kvm/vmx/nested.c | 1 -

 1 file changed, 1 deletion(-)
arch/x86/kvm/vmx/nested.c +0 -0
index 778f82015f03..bfa366938c49 100644
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0

                                  SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | SECONDARY_EXEC_APIC_REGISTER_VIRT | SECONDARY_EXEC_ENABLE_VMFUNC | -                                 SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_DESC);

                if (nested_cpu_has(vmcs12,

#2 lore raw
Quoted 6 lines · Maxim Levitsky
Quoted 5 lines · Vitaly Kuznetsov

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency

^^^ scaling ^^^

Quoted 11 lines
Quoted 4 lines

for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.

Ouch.

I guess that needs a Fixes tag?

Fixes d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")

I dismissed that because prior to d041b5ea93352b SECONDARY_EXEC_TSC_SCALING was filtered out in nested_vmx_setup_ctls_msrs() but now I think I was wrong, SECONDARY_EXEC_TSC_SCALING was likely kept in VMCS02 regardless of that. Will add in v2.

Quoted 5 lines

Also this is thankfully Intel specific, because in AMD you can't enable TSC scaling - there is just an MSR with default value of 1.0, which one can change if TSC scaling is supported in CPUID.

Thanks!

Quoted 28 lines

Best regards,
Maxim Levitsky

Quoted 22 lines

Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when it was set for L1. TSC_MULTIPLIER is already correctly computed and written by prepare_vmcs02().

---
 arch/x86/kvm/vmx/nested.c | 1 -

 1 file changed, 1 deletion(-)
arch/x86/kvm/vmx/nested.c +0 -0
index 778f82015f03..bfa366938c49 100644
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0

                                  SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | SECONDARY_EXEC_APIC_REGISTER_VIRT | SECONDARY_EXEC_ENABLE_VMFUNC | -                                 SECONDARY_EXEC_TSC_SCALING | SECONDARY_EXEC_DESC);

                if (nested_cpu_has(vmcs12,

Signature · 1 line
Vitaly
#3 lore raw
Quoted 20 lines · Vitaly Kuznetsov
Quoted 6 lines · Maxim Levitsky
Quoted 5 lines · Vitaly Kuznetsov

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is supported. The problem appears to be that Windows doesn't use TSC frequency

^^^ scaling ^^^

Quoted 9 lines
Quoted 4 lines

for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.

Ouch.

I guess that needs a Fixes tag?

Quoted 7 lines
Quoted 2 lines
Fixes d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")

I dismissed that because prior to d041b5ea93352b SECONDARY_EXEC_TSC_SCALING was filtered out in nested_vmx_setup_ctls_msrs() but now I think I was wrong, SECONDARY_EXEC_TSC_SCALING was likely kept in VMCS02 regardless of that. Will add in v2.

Yep, it would have been kept in vmcs02 even though the feature wasn't advertised to the L1 VMM. A Cc for stable is warranted as well.

I added this (with the tags and s/frequency/scaling) to the queue of patches for 5.20 I have lined up for Paolo to consume on his return. Paolo and I haven't hashed out how we'll actually manage anything, i.e. my list is speculative, but unless you hear otherwise, no need to send a v2.

#4 lore raw
In reply to Vitaly Kuznetsov

Hi Vitaly,

Quoted 3 lines · Vitaly Kuznetsov

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is

Would you mind helping clarify if it is L1 or L2 that hangs?

The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.

Thank you very much!

Dongli Zhang

Quoted 28 lines

supported. The problem appears to be that Windows doesn't use TSC frequency for its guests even when the feature is advertised and KVM filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from L1's. This leads to L2 running with the default frequency (matching host's) while L1 is running with an altered one.

Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when it was set for L1. TSC_MULTIPLIER is already correctly computed and written by prepare_vmcs02().

---
 arch/x86/kvm/vmx/nested.c | 1 -
 1 file changed, 1 deletion(-)
arch/x86/kvm/vmx/nested.c +0 -1
index 778f82015f03..bfa366938c49 100644
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
 				  SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
 				  SECONDARY_EXEC_APIC_REGISTER_VIRT |
 				  SECONDARY_EXEC_ENABLE_VMFUNC |
-				  SECONDARY_EXEC_TSC_SCALING |
 				  SECONDARY_EXEC_DESC);
 
 		if (nested_cpu_has(vmcs12,
#5 lore raw
Quoted 12 lines · Dongli Zhang

Hi Vitaly,

Quoted 3 lines · Vitaly Kuznetsov

Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to hang upon boot or shortly after when a non-default TSC frequency was set for L1. The issue is observed on a host where TSC scaling is

Would you mind helping clarify if it is L1 or L2 that hangs?

The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.

I think it's L2 but I'm not sure: there's no easy way to interract with L1 (Hyper-V) directly, all the interfaces (UI, network,..) are handled by L2 (Windows). Prior to the observed 'hang' Hyper-V (L1) programms synthetic timer in KVM too far in the future but my guess is that it's doing that on Windows' (L2) behalf, basically just relaying the request. The issue only shows up with 'hv-reenlightenment' + 'hv-frequencies' (in QEMU's terms) features as in this case both Hyper-V (L1) and Windows (L2) trust the information about TSC frequency from KVM but the information turns out to be incorrect for Windows (L2).

Signature · 1 line
Vitaly