Skip to contentSkip to conversation

x86/CPU/AMD: Clear virtualized VMLOAD/VMSAVE on Zen4 client

PATCH In review Mario Limonciello 3 participants 8 messages Nov 5, 2024 – Nov 6, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/CPU/AMD: Clear virtualized VMLOAD/VMSAVE on Zen4 client

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
 arch/x86/kernel/cpu/amd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/amd.c +11 -0
index 015971adadfc7..ecd42c2b3242e 100644
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
 {
 	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
 		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.
+	 */
+	switch (c->x86_model) {
+	case 0x18 ... 0x1f:
+	case 0x60 ... 0x7f:
+		clear_cpu_cap(c, X86_FEATURE_V_VMSAVE_VMLOAD);
+		break;
+	}
 }
 
 static void init_amd_zen5(struct cpuinfo_x86 *c)
Signature · 1 line
2.43.0
#1 lore raw

+KVM, given that this quite obviously affects KVM...

Quoted 21 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
 arch/x86/kernel/cpu/amd.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/amd.c +0 -0
index 015971adadfc7..ecd42c2b3242e 100644

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 7 lines
arch/x86/kernel/cpu/amd.c +4 -0
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
 {
 	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
 		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 13 lines
arch/x86/kernel/cpu/amd.c +7 -0
+	 */
+	switch (c->x86_model) {
+	case 0x18 ... 0x1f:
+	case 0x60 ... 0x7f:
+		clear_cpu_cap(c, X86_FEATURE_V_VMSAVE_VMLOAD);
+		break;
+	}
 }
 
 static void init_amd_zen5(struct cpuinfo_x86 *c)
Signature · 1 line
2.43.0
#2 lore raw
Quoted 19 lines · Sean Christopherson

+KVM, given that this quite obviously affects KVM...

Quoted 16 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
  arch/x86/kernel/cpu/amd.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
Quoted 14 lines

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 12 lines
arch/x86/kernel/cpu/amd.c +4 -0
index 015971adadfc7..ecd42c2b3242e 100644
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
  {
  	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
  		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

BIOS bug. Those shouldn't have been advertised.

Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/cpu/amd.c +7 -0
+	 */
+	switch (c->x86_model) {
+	case 0x18 ... 0x1f:
+	case 0x60 ... 0x7f:
+		clear_cpu_cap(c, X86_FEATURE_V_VMSAVE_VMLOAD);
+		break;
+	}
  }
 
  static void init_amd_zen5(struct cpuinfo_x86 *c)
Signature · 1 line
2.43.0
#3 lore raw
Quoted 35 lines · Mario Limonciello
Quoted 19 lines · Sean Christopherson

+KVM, given that this quite obviously affects KVM...

Quoted 16 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
  arch/x86/kernel/cpu/amd.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
Quoted 14 lines

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 12 lines
arch/x86/kernel/cpu/amd.c +4 -0
index 015971adadfc7..ecd42c2b3242e 100644
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
  {
  	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
  		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 1 line

BIOS bug. Those shouldn't have been advertised.

Why not? "but they can lead to random host reboots" is a description of the symptom, not an explanation for why KVM is unable to use a feature that is apparently support by the CPU.

And if the CPU doesn't actually support virtualized VMLOAD/VMSAVE, then this is a much bigger problem, because it means KVM is effectively giving the guest read and write access to all of host memory.

#4 lore raw
Quoted 36 lines · Sean Christopherson
Quoted 35 lines · Mario Limonciello
Quoted 33 lines · Sean Christopherson

+KVM, given that this quite obviously affects KVM...

Quoted 21 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
  arch/x86/kernel/cpu/amd.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/amd.c +0 -0
index 015971adadfc7..ecd42c2b3242e 100644

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 7 lines
arch/x86/kernel/cpu/amd.c +4 -0
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
  {
  	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
  		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 1 line
Quoted 1 line

BIOS bug. Those shouldn't have been advertised.

Hi!

My question is, why would AMD drop support intentionally for VLS on client machines?

I understand that there might be a errata, and I don't object disabling the feature because of this.

But hearing that 'These instructions aren't intended to be advertised' means that AMD intends to stop supporting virtualization on client systems or at least partially do so.

That worries me. So far AMD was much better that Intel supporting most of the features across all of the systems which is very helpful in various scenarios, and this is very appreciated by the community.

Speaking strictly personally here, as a AMD fan.

Best regards,
Maxim Levitsky

Quoted 9 lines

Why not? "but they can lead to random host reboots" is a description of the symptom, not an explanation for why KVM is unable to use a feature that is apparently support by the CPU.

And if the CPU doesn't actually support virtualized VMLOAD/VMSAVE, then this is a much bigger problem, because it means KVM is effectively giving the guest read and write access to all of host memory.

#5 lore raw
Quoted 41 lines · Maxim Levitsky
Quoted 36 lines · Sean Christopherson
Quoted 35 lines · Mario Limonciello
Quoted 33 lines · Sean Christopherson

+KVM, given that this quite obviously affects KVM...

Quoted 21 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
   arch/x86/kernel/cpu/amd.c | 11 +++++++++++
   1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/amd.c +0 -0
index 015971adadfc7..ecd42c2b3242e 100644

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 7 lines
arch/x86/kernel/cpu/amd.c +4 -0
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
   {
   	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
   		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 1 line
Quoted 1 line

BIOS bug. Those shouldn't have been advertised.

Hi!

Quoted 8 lines

My question is, why would AMD drop support intentionally for VLS on client machines?

I understand that there might be a errata, and I don't object disabling the feature because of this.

But hearing that 'These instructions aren't intended to be advertised' means that AMD intends to stop supporting virtualization on client systems or at least partially do so.

Don't read into it too far. It's just a BIOS problem with those instructions "specifically" on the processors indicated here. Other processors (for example Zen 5 client processors) do correctly advertise support where applicable.

When they launched those bits weren't supposed to be set to indicate support, but BIOS did set them.

Quoted 21 lines

That worries me. So far AMD was much better that Intel supporting most of the features across all of the systems which is very helpful in various scenarios, and this is very appreciated by the community.

Speaking strictly personally here, as a AMD fan.

Best regards,> Maxim Levitsky

Quoted 9 lines

Why not? "but they can lead to random host reboots" is a description of the symptom, not an explanation for why KVM is unable to use a feature that is apparently support by the CPU.

And if the CPU doesn't actually support virtualized VMLOAD/VMSAVE, then this is a much bigger problem, because it means KVM is effectively giving the guest read and write access to all of host memory.

I'm gathering that what supported means to you and what it means to me are different things. "Architecturally" the instructions for virtualized VMLOAD/VMSAVE exist. There are problems with them on these processors, and for that reason the BIOS was not supposed to set those bits but it did.

#6 lore raw
Quoted 4 lines · Mario Limonciello
Quoted 3 lines · Maxim Levitsky
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Mario Limonciello

On 11/6/2024 09:03, Sean Christopherson wrote:

...

Quoted 28 lines
Quoted 19 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 12 lines
arch/x86/kernel/cpu/amd.c +4 -0
index 015971adadfc7..ecd42c2b3242e 100644
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
   {
   	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
   		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 1 line
Quoted 1 line

BIOS bug. Those shouldn't have been advertised.

Hi!

Quoted 8 lines

My question is, why would AMD drop support intentionally for VLS on client machines?

I understand that there might be a errata, and I don't object disabling the feature because of this.

But hearing that 'These instructions aren't intended to be advertised' means that AMD intends to stop supporting virtualization on client systems or at least partially do so.

Quoted 7 lines

Don't read into it too far. It's just a BIOS problem with those instructions "specifically" on the processors indicated here. Other processors (for example Zen 5 client processors) do correctly advertise support where applicable.

When they launched those bits weren't supposed to be set to indicate support, but BIOS did set them.

As you quite clearly call out below, this isn't simply a BIOS problem.

Quoted 21 lines
Quoted 4 lines

That worries me. So far AMD was much better that Intel supporting most of the features across all of the systems which is very helpful in various scenarios, and this is very appreciated by the community.

Quoted 16 lines

Speaking strictly personally here, as a AMD fan.

Best regards,> Maxim Levitsky

Quoted 9 lines

Why not? "but they can lead to random host reboots" is a description of the symptom, not an explanation for why KVM is unable to use a feature that is apparently support by the CPU.

And if the CPU doesn't actually support virtualized VMLOAD/VMSAVE, then this is a much bigger problem, because it means KVM is effectively giving the guest read and write access to all of host memory.

Quoted 2 lines

I'm gathering that what supported means to you and what it means to me are different things.

Yes. And the distinction matters greatly in this case, because "VMLOAD/VMSAVE in the guest are broken" is *very* different than "VMLOAD/VMSAVE in the guest actually operate on SPAs, not GPAs".

Quoted 1 line

"Architecturally" the instructions for virtualized VMLOAD/VMSAVE exist.

Which means they're supported, but broken.

Quoted 2 lines

There are problems with them on these processors, and for that reason the BIOS was not supposed to set those bits but it did.

In other words, this a CPU bug. The kernel comment absolutely needs to reflect that. Passing this off as BIOS going rogue is misleading and confusing.

#7 lore raw
In reply to Mario Limonciello
Quoted 47 lines · Mario Limonciello
Quoted 46 lines · Maxim Levitsky
Quoted 36 lines · Sean Christopherson
Quoted 35 lines · Mario Limonciello
Quoted 33 lines · Sean Christopherson

+KVM, given that this quite obviously affects KVM...

Quoted 21 lines · Mario Limonciello

From: Mario Limonciello <[email protected]>

A number of Zen4 client SoCs advertise the ability to use virtualized VMLOAD/VMSAVE, but using these instructions is reported to be a cause of a random host reboot.

These instructions aren't intended to be advertised on Zen4 client so clear the capability.

---
   arch/x86/kernel/cpu/amd.c | 11 +++++++++++
   1 file changed, 11 insertions(+)
arch/x86/kernel/cpu/amd.c +0 -0
index 015971adadfc7..ecd42c2b3242e 100644

arch/x86/kernel/cpu/amd.c static void init_amd_zen4(struct cpuinfo_x86 *c) · line 930

Quoted 7 lines
arch/x86/kernel/cpu/amd.c +4 -0
@@ -924,6 +924,17 @@ static void init_amd_zen4(struct cpuinfo_x86 *c)
   {
   	if (!cpu_has(c, X86_FEATURE_HYPERVISOR))
   		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_SHARED_BTB_FIX_BIT);
+
+	/*
+	 * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+	 * in some BIOS versions but they can lead to random host reboots.

Uh, CPU bug? Erratum?

Quoted 1 line
Quoted 1 line

BIOS bug. Those shouldn't have been advertised.

Hi!

My question is, why would AMD drop support intentionally for VLS on client machines?

I understand that there might be a errata, and I don't object disabling the feature because of this.

Quoted 8 lines
Quoted 3 lines

But hearing that 'These instructions aren't intended to be advertised' means that AMD intends to stop supporting virtualization on client systems or at least partially do so.

Don't read into it too far. It's just a BIOS problem with those instructions "specifically" on the processors indicated here. Other processors (for example Zen 5 client processors) do correctly advertise support where applicable.

I am very glad to hear that, thanks!

Quoted 3 lines

When they launched those bits weren't supposed to be set to indicate support, but BIOS did set them.

In other words if I understand correctly, there was an errata and to work it around on the affected CPUs, AMD decided to disable the feature in CPUID, which is reasonable, but some BIOS vendors forgot to do this.

It all makes sense, thanks again!

Best regards,
Maxim Levitsky

Quoted 24 lines
Quoted 17 lines

That worries me. So far AMD was much better that Intel supporting most of the features across all of the systems which is very helpful in various scenarios, and this is very appreciated by the community.

Speaking strictly personally here, as a AMD fan.

Best regards,> Maxim Levitsky

Quoted 8 lines

Why not? "but they can lead to random host reboots" is a description of the symptom, not an explanation for why KVM is unable to use a feature that is apparently support by the CPU.

And if the CPU doesn't actually support virtualized VMLOAD/VMSAVE, then this is a much bigger problem, because it means KVM is effectively giving the guest read and write access to all of host memory.

I'm gathering that what supported means to you and what it means to me are different things. "Architecturally" the instructions for virtualized VMLOAD/VMSAVE exist. There are problems with them on these processors, and for that reason the BIOS was not supposed to set those bits but it did.