Skip to contentSkip to conversation

AMD nested: commit broke VMware

Bug [email protected] 4 participants 9 messages May 29, 2023 – Aug 1, 2023 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello,
Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs.

User reports are here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583
https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/

I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"

I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer.

First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed:
https://lore.kernel.org/kvm/[email protected]/

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

#1 lore raw

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 10 lines

Hello,
Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs.

User reports are here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583
https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/

I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"

Quoted 8 lines

I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer.

First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed:
https://lore.kernel.org/kvm/[email protected]/

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends
(just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

I hope that I will find time to do this soon.

Best regards,
Maxim Levitsky

#2 lore raw
Quoted 21 lines · Maxim Levitsky

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 10 lines

Hello,
Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs.

User reports are here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583
https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/

I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"

Quoted 8 lines

I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer.

First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed:
https://lore.kernel.org/kvm/[email protected]/

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

Quoted 6 lines

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends
(just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

Quoted 9 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

I hope that I will find time to do this soon.

Best regards,
Maxim Levitsky

#3 lore raw
Quoted 27 lines · Jim Mattson
Quoted 25 lines · Maxim Levitsky

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 10 lines

Hello,
Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs.

User reports are here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583
https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/

I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"

Quoted 8 lines

I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer.

First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed:
https://lore.kernel.org/kvm/[email protected]/

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends
(just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 6 lines
Quoted 2 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Quoted 9 lines
Quoted 9 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

I hope that I will find time to do this soon.

Best regards,
Maxim Levitsky

#4 lore raw

У вт, 2023-05-30 у 13:34 -0700, Jim Mattson пише:

Quoted 42 lines · Jim Mattson
Quoted 25 lines · Maxim Levitsky
Quoted 24 lines

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 10 lines

Hello,
Since kernel 5.16 users can't start VMware VMs when it is nested under KVM on AMD CPUs.

User reports are here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2008583
https://forums.unraid.net/topic/128868-vmware-7x-will-not-start-any-vms-under-unraid-6110/

I've pinpointed it to commit 174a921b6975ef959dd82ee9e8844067a62e3ec1 (appeared in 5.16rc1) "nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"

Quoted 8 lines

I've confirmed that VMware errors out when it checks for TLB_CONTROL_FLUSH_ASID support and gets a 'false' answer.

First revisions of the patch in question had some support for TLB_CONTROL_FLUSH_ASID, but it was removed:
https://lore.kernel.org/kvm/[email protected]/

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends
(just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 6 lines
Quoted 2 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

Quoted 15 lines

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Quoted 9 lines
Quoted 9 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

I hope that I will find time to do this soon.

Best regards,
Maxim Levitsky

Yup...

After applying this horrible hack to KVM, the VM still boots just fine on bare metal.

arch/x86/include/asm/svm.h +4 -4
index e7c7379d6ac7b0..2e45c1b747104a 100644
@@ -170,10 +170,10 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 };
 
 
-#define TLB_CONTROL_DO_NOTHING 0
-#define TLB_CONTROL_FLUSH_ALL_ASID 1
-#define TLB_CONTROL_FLUSH_ASID 3
-#define TLB_CONTROL_FLUSH_ASID_LOCAL 7
+#define TLB_CONTROL_DO_NOTHING 0xF0
+#define TLB_CONTROL_FLUSH_ALL_ASID 0xF1
+#define TLB_CONTROL_FLUSH_ASID 0xF3
+#define TLB_CONTROL_FLUSH_ASID_LOCAL 0xF7
 
 #define V_TPR_MASK 0x0f
 

Shall we revert the offending patch then?

Best regards,
Maxim Levitsky

#5 lore raw
Quoted 36 lines · Maxim Levitsky

У вт, 2023-05-30 у 13:34 -0700, Jim Mattson пише:

Quoted 35 lines · Jim Mattson
Quoted 12 lines · Maxim Levitsky
Quoted 11 lines

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 4 lines

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends (just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 7 lines
Quoted 3 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Ya. The APM even provides a definition of "reserved" that explicitly calls out the different reserved qualifiers. The only fields/values that KVM can actively enforce are things tagged MBZ.

reserved
Fields marked as reserved may be used at some future time.
To preserve compatibility with future processors, reserved fields require special handling when read or written by software. Software must not depend on the state of a reserved field (unless qualified as RAZ), nor upon the ability of such fields to return a previously written state. If a field is marked reserved without qualification, software must not change the state of that field; it must reload that field with the same value returned from a prior read.
Reserved fields may be qualified as IGN, MBZ, RAZ, or SBZ (see definitions).

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

I hope that I will find time to do this soon.

...

Quoted 1 line

Shall we revert the offending patch then?

Yes please.

#6 lore raw
Quoted 55 lines · Sean Christopherson
Quoted 36 lines · Maxim Levitsky

У вт, 2023-05-30 у 13:34 -0700, Jim Mattson пише:

Quoted 35 lines · Jim Mattson
Quoted 12 lines · Maxim Levitsky
Quoted 11 lines

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 4 lines

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends (just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 7 lines
Quoted 3 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Ya. The APM even provides a definition of "reserved" that explicitly calls out the different reserved qualifiers. The only fields/values that KVM can actively enforce are things tagged MBZ.

reserved
Fields marked as reserved may be used at some future time.
To preserve compatibility with future processors, reserved fields require special handling when read or written by software. Software must not depend on the state of a reserved field (unless qualified as RAZ), nor upon the ability of such fields to return a previously written state. If a field is marked reserved without qualification, software must not change the state of that field; it must reload that field with the same value returned from a prior read.
Reserved fields may be qualified as IGN, MBZ, RAZ, or SBZ (see definitions).

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I hope that I will find time to do this soon.

...

Quoted 1 line

Shall we revert the offending patch then?

Yes please.

It's not quite that simple.

The vmcb12 TLB_CONTROL field needs to be sanitized on its way into the vmcb02 (perhaps in nested_copy_vmcb_control_to_cache()?). Bits 63:2 should be cleared. Also, if the guest CPUID does not advertise support for FlushByASID, then bit 1 should be cleared. Note that the vmcb12 TLB_CONTROL field itself must not be modified, since the APM specifically states, "The VMRUN instruction reads, but does not change, the value of the TLB_CONTROL field."

#8 lore raw
In reply to a message not in the archive

Hi
Are there any thoughts on this?

PS As I see it, that commit didn't fix any real problem (up to 5.15 nothing was broken that required it), but did break nested VMWare Workstation/ESXi for people that use it (yes, it's vmware's code bug, but doubt they will fix it).

May 31, 2023, 15:34 by [email protected]:

Quoted 76 lines · Sean Christopherson
Quoted 56 lines · Maxim Levitsky
Quoted 36 lines

У вт, 2023-05-30 у 13:34 -0700, Jim Mattson пише:

Quoted 35 lines · Jim Mattson
Quoted 12 lines · Maxim Levitsky
Quoted 11 lines

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 4 lines

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends (just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 7 lines
Quoted 3 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Ya. The APM even provides a definition of "reserved" that explicitly calls out the different reserved qualifiers. The only fields/values that KVM can actively enforce are things tagged MBZ.

reserved
Fields marked as reserved may be used at some future time.
To preserve compatibility with future processors, reserved fields require special handling when read or written by software. Software must not depend on the state of a reserved field (unless qualified as RAZ), nor upon the ability of such fields to return a previously written state. If a field is marked reserved without qualification, software must not change the state of that field; it must reload that field with the same value returned from a prior read.
Reserved fields may be qualified as IGN, MBZ, RAZ, or SBZ (see definitions).

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I hope that I will find time to do this soon.

...

Quoted 1 line

Shall we revert the offending patch then?

Yes please.

It's not quite that simple.

The vmcb12 TLB_CONTROL field needs to be sanitized on its way into the vmcb02 (perhaps in nested_copy_vmcb_control_to_cache()?). Bits 63:2 should be cleared. Also, if the guest CPUID does not advertise support for FlushByASID, then bit 1 should be cleared. Note that the vmcb12 TLB_CONTROL field itself must not be modified, since the APM specifically states, "The VMRUN instruction reads, but does not change, the value of the TLB_CONTROL field."

#9 lore raw

У вт, 2023-08-01 у 10:49 +0200, [email protected] пише:

Quoted 83 lines

Hi
Are there any thoughts on this?

PS As I see it, that commit didn't fix any real problem (up to 5.15 nothing was broken that required it), but did break nested VMWare Workstation/ESXi for people that use it (yes, it's vmware's code bug, but doubt they will fix it).

May 31, 2023, 15:34 by [email protected]:

Quoted 75 lines · Sean Christopherson
Quoted 55 lines · Maxim Levitsky
Quoted 36 lines

У вт, 2023-05-30 у 13:34 -0700, Jim Mattson пише:

Quoted 35 lines · Jim Mattson
Quoted 12 lines · Maxim Levitsky
Quoted 11 lines

У пн, 2023-05-29 у 14:58 +0200, [email protected] пише:

Quoted 4 lines

I don't know what would be the best case here, maybe put a quirk there, so it doesn't break "userspace". Committer's email is dead, so I'm writing here.

I have to say that I know about this for long time, because some time ago I used to play with VMware player in a VM on AMD on my spare time, on weekends (just doing various crazy things with double nesting, running win98 nested, vfio stuff, etc, etc).

Quoted 7 lines
Quoted 3 lines

I didn't report it because its a bug in VMWARE - they set a bit in the tlb_control without checking CPUID's FLUSHBYASID which states that KVM doesn't support setting this bit.

I am pretty sure that bit 1 is supposed to be ignored on hardware without FlushByASID, but I'll have to see if I can dig up an old APM to verify that.

I couldn't find an APM that old, but even today's APM does not specify that any checks are performed on the TLB_CONTROL field by VMRUN.

While Intel likes to fail VM-entry for illegal VMCS state, AMD prefers to massage the VMCB to render any illegal VMCB state legal. For example, rather than fail VM-entry for a non-canonical address, AMD is inclined to drop the high bits and sign-extend the low bits, so that the address is canonical.

I'm willing to bet that modern CPUs continue to ignore the TLB_CONTROL bits that were noted "reserved" in version 3.22 of the manual, and that Krish simply manufactured the checks in commit 174a921b6975 ("nSVM: Check for reserved encodings of TLB_CONTROL in nested VMCB"), without cause.

Ya. The APM even provides a definition of "reserved" that explicitly calls out the different reserved qualifiers. The only fields/values that KVM can actively enforce are things tagged MBZ.

reserved
Fields marked as reserved may be used at some future time.
To preserve compatibility with future processors, reserved fields require special handling when read or written by software. Software must not depend on the state of a reserved field (unless qualified as RAZ), nor upon the ability of such fields to return a previously written state. If a field is marked reserved without qualification, software must not change the state of that field; it must reload that field with the same value returned from a prior read.
Reserved fields may be qualified as IGN, MBZ, RAZ, or SBZ (see definitions).

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Supporting FLUSHBYASID would fix this, and make nesting faster too, but it is far from a trivial job.

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I hope that I will find time to do this soon.

...

Quoted 1 line

Shall we revert the offending patch then?

Yes please.

It's not quite that simple.

The vmcb12 TLB_CONTROL field needs to be sanitized on its way into the vmcb02 (perhaps in nested_copy_vmcb_control_to_cache()?). Bits 63:2 should be cleared. Also, if the guest CPUID does not advertise support for FlushByASID, then bit 1 should be cleared. Note that the vmcb12 TLB_CONTROL field itself must not be modified, since the APM specifically states, "The VMRUN instruction reads, but does not change, the value of the TLB_CONTROL field."

I will prepare a patch for this hopefully this week. Jim mentioned that a revert is not full solution for this.

Best regards,
Maxim Levitsky