Skip to contentSkip to conversation

KVM: SVM: reduce guest MAXPHYADDR by one in case C-bit is a physical bit

RFC PATCH In review Vitaly Kuznetsov 4 participants 7 messages Oct 15, 2021 – Oct 18, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 KVM: SVM: reduce guest MAXPHYADDR by one in case C-bit is a physical bit

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000 Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false
pid=47131 tid=47134 errno=0 - Success

      1	0x000000000040281b: vcpu_worker at demand_paging_test.c:63
      2	0x00007fb36716e431: ?? ??:0
      3	0x00007fb36709c912: ?? ??:0

Invalid guest sync status: exit_reason=SHUTDOWN

The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:

Address sizes: 43 bits physical, 48 bits virtual

The guest physical address space, however, is not reduced as stated in commit e39f00f60ebd ("KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):

Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.

Implement the reduction.

Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)

---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.

---
 arch/x86/kvm/cpuid.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
arch/x86/kvm/cpuid.c +10 -3
index 751aa85a3001..04ae280a0b66 100644
@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 *
 		 * If TDP is enabled but an explicit guest MAXPHYADDR is not
 		 * provided, use the raw bare metal MAXPHYADDR as reductions to
-		 * the HPAs do not affect GPAs.
+		 * the HPAs do not affect GPAs. The value, however, has to be
+		 * reduced by 1 in case C-bit is a physical bit (APM section
+		 * 15.34.6).
 		 */
-		if (!tdp_enabled)
+		if (!tdp_enabled) {
 			g_phys_as = boot_cpu_data.x86_phys_bits;
-		else if (!g_phys_as)
+		} else if (!g_phys_as) {
 			g_phys_as = phys_as;
 
+			if (kvm_cpu_cap_has(X86_FEATURE_SEV) &&
+			    (cpuid_ebx(0x8000001f) & 0x3f) < g_phys_as)
+				g_phys_as -= 1;
+		}
+
 		entry->eax = g_phys_as | (virt_as << 8);
 		entry->edx = 0;
 		cpuid_entry_override(entry, CPUID_8000_0008_EBX);
Signature · 1 line
2.31.1
#1 lore raw
Quoted 8 lines · Vitaly Kuznetsov

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000

This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]

Quoted 51 lines

Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success

      1	0x000000000040281b: vcpu_worker at demand_paging_test.c:63
      2	0x00007fb36716e431: ?? ??:0
      3	0x00007fb36709c912: ?? ??:0

Invalid guest sync status: exit_reason=SHUTDOWN

The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:

Address sizes: 43 bits physical, 48 bits virtual

The guest physical address space, however, is not reduced as stated in commit e39f00f60ebd ("KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):

Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.

Implement the reduction.

Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)

---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.

---
 arch/x86/kvm/cpuid.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
arch/x86/kvm/cpuid.c +3 -1
index 751aa85a3001..04ae280a0b66 100644
@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 *
 		 * If TDP is enabled but an explicit guest MAXPHYADDR is not
 		 * provided, use the raw bare metal MAXPHYADDR as reductions to
-		 * the HPAs do not affect GPAs.
+		 * the HPAs do not affect GPAs. The value, however, has to be
+		 * reduced by 1 in case C-bit is a physical bit (APM section
+		 * 15.34.6).
 		 */

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937

Quoted 10 lines
arch/x86/kvm/cpuid.c +5 -2
-		if (!tdp_enabled)
+		if (!tdp_enabled) {
 			g_phys_as = boot_cpu_data.x86_phys_bits;
-		else if (!g_phys_as)
+		} else if (!g_phys_as) {
 			g_phys_as = phys_as;
 
+			if (kvm_cpu_cap_has(X86_FEATURE_SEV) &&
+			    (cpuid_ebx(0x8000001f) & 0x3f) < g_phys_as)
+				g_phys_as -= 1;

This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]

[*] https://lkml.kernel.org/r/[email protected]

Quoted 8 lines
arch/x86/kvm/cpuid.c +2 -0
+		}
+
 		entry->eax = g_phys_as | (virt_as << 8);
 		entry->edx = 0;
 		cpuid_entry_override(entry, CPUID_8000_0008_EBX);
Signature · 1 line
2.31.1
#2 lore raw
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

Quoted 7 lines
Quoted 3 lines

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000

This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].

Hi Vitaly and everyone!

You are the 3rd person to suffer from this issue :-( Sean Christopherson was first, I was second.

I reported this, then I think we found out that it is not the HyperTransport region after all, and I think that the whole thing lost in 'trying to get answers from AMD'.

https://lore.kernel.org/lkml/[email protected]/

I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.

Sorry that you had to debug this.

Best regards,
Maxim Levitsky

Quoted 80 lines

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]

Quoted 51 lines

Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success

      1	0x000000000040281b: vcpu_worker at demand_paging_test.c:63
      2	0x00007fb36716e431: ?? ??:0
      3	0x00007fb36709c912: ?? ??:0

Invalid guest sync status: exit_reason=SHUTDOWN

The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:

Address sizes: 43 bits physical, 48 bits virtual

The guest physical address space, however, is not reduced as stated in commit e39f00f60ebd ("KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):

Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.

Implement the reduction.

Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)

---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.

---
 arch/x86/kvm/cpuid.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
arch/x86/kvm/cpuid.c +3 -1
index 751aa85a3001..04ae280a0b66 100644
@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 *
 		 * If TDP is enabled but an explicit guest MAXPHYADDR is not
 		 * provided, use the raw bare metal MAXPHYADDR as reductions to
-		 * the HPAs do not affect GPAs.
+		 * the HPAs do not affect GPAs. The value, however, has to be
+		 * reduced by 1 in case C-bit is a physical bit (APM section
+		 * 15.34.6).
 		 */

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937

Quoted 10 lines
arch/x86/kvm/cpuid.c +5 -2
-		if (!tdp_enabled)
+		if (!tdp_enabled) {
 			g_phys_as = boot_cpu_data.x86_phys_bits;
-		else if (!g_phys_as)
+		} else if (!g_phys_as) {
 			g_phys_as = phys_as;
 
+			if (kvm_cpu_cap_has(X86_FEATURE_SEV) &&
+			    (cpuid_ebx(0x8000001f) & 0x3f) < g_phys_as)
+				g_phys_as -= 1;

This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]

[*] https://lkml.kernel.org/r/[email protected]

Quoted 8 lines
arch/x86/kvm/cpuid.c +2 -0
+		}
+
 		entry->eax = g_phys_as | (virt_as << 8);
 		entry->edx = 0;
 		cpuid_entry_override(entry, CPUID_8000_0008_EBX);
Signature · 1 line
2.31.1
#3 lore raw
Quoted 22 lines · Maxim Levitsky
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

Quoted 7 lines
Quoted 3 lines

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000

This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].

Hi Vitaly and everyone!

You are the 3rd person to suffer from this issue :-( Sean Christopherson was first, I was second.

I reported this, then I think we found out that it is not the HyperTransport region after all, and I think that the whole thing lost in 'trying to get answers from AMD'.

Quoted 7 lines

https://lore.kernel.org/lkml/[email protected]/

I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.

Sorry that you had to debug this.

I didn't spend too much time on this, that's the reson for 'RFC' :-) I agree we need at least a short-term solution as permanently failing tests may start masking newly introduces issues.

Signature · 1 line
Vitaly
maintainer
#4 lore raw
In reply to Maxim Levitsky
Quoted 2 lines · Maxim Levitsky

I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.

What's going on is documented in the thread at
https://yhbt.net/lore/all/[email protected]/:

Quoted 9 lines

That doesn't really follow what Andrew gave us, namely:

1) On parts with <40 bits, its fully hidden from software

2) Before Fam17h, it was always 12G just below 1T, even if there was more RAM above this location

3) On Fam17h and later, it is variable based on SME, and is either just below 2^48 (no encryption) or 2^43 (encryption)

If you can use this information to implement the fix, that'd be very nice. I didn't apply the hackish fix because I wanted to test it on a SME-enabled box.

Paolo

#5 lore raw
In reply to Sean Christopherson
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Vitaly Kuznetsov

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000

Quoted 5 lines

This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].

[1] https://lore.kernel.org/lkml/[email protected]/

Hey,

it seems I'm only three months late to the party!

Quoted 3 lines

[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]

This patch helps indeed, thanks! Paolo, any particular reason you haven't queued it yet?

Quoted 58 lines
Quoted 58 lines

Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success

      1	0x000000000040281b: vcpu_worker at demand_paging_test.c:63
      2	0x00007fb36716e431: ?? ??:0
      3	0x00007fb36709c912: ?? ??:0

Invalid guest sync status: exit_reason=SHUTDOWN

The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:

Address sizes: 43 bits physical, 48 bits virtual

The guest physical address space, however, is not reduced as stated in commit e39f00f60ebd ("KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):

Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.

Implement the reduction.

Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)

---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.

---
 arch/x86/kvm/cpuid.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
arch/x86/kvm/cpuid.c +5 -3
index 751aa85a3001..04ae280a0b66 100644
@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 *
 		 * If TDP is enabled but an explicit guest MAXPHYADDR is not
 		 * provided, use the raw bare metal MAXPHYADDR as reductions to
-		 * the HPAs do not affect GPAs.
+		 * the HPAs do not affect GPAs. The value, however, has to be
+		 * reduced by 1 in case C-bit is a physical bit (APM section
+		 * 15.34.6).
 		 */
-		if (!tdp_enabled)
+		if (!tdp_enabled) {
 			g_phys_as = boot_cpu_data.x86_phys_bits;
-		else if (!g_phys_as)
+		} else if (!g_phys_as) {
 			g_phys_as = phys_as;
 
Quoted 9 lines

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937

Quoted 3 lines
arch/x86/kvm/cpuid.c +3 -0
+			if (kvm_cpu_cap_has(X86_FEATURE_SEV) &&
+			    (cpuid_ebx(0x8000001f) & 0x3f) < g_phys_as)
+				g_phys_as -= 1;

This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]

[*] https://lkml.kernel.org/r/[email protected]

I see, thanks for the pointer.

Quoted 9 lines
Quoted 8 lines
arch/x86/kvm/cpuid.c +2 -0
+		}
+
 		entry->eax = g_phys_as | (virt_as << 8);
 		entry->edx = 0;
 		cpuid_entry_override(entry, CPUID_8000_0008_EBX);
Signature · 1 line
2.31.1
Signature · 1 line
Vitaly
#6 lore raw
Quoted 20 lines · Vitaly Kuznetsov
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Vitaly Kuznetsov

Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:

./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000

Quoted 5 lines

This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].

[1] https://lore.kernel.org/lkml/[email protected]/

Hey,

Quoted 7 lines

it seems I'm only three months late to the party!

Quoted 3 lines

[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]

This patch helps indeed

FWIW, 'access_tracking_perf_test' remains broken even after the patch is applied:

# ./access_tracking_perf_test 
Testing guest mode: PA-bits:ANY, VA-bits:48,  4K pages
guest physical test memory offset: 0xfffcbffff000

Populating memory : 3.858448918s
Writing to populated memory : 0.937319626s
Reading from populated memory : 0.003073207s ==== Test Assertion Failure ====
lib/kvm_util.c:1382: false
pid=6422 tid=6425 errno=4 - Interrupted system call

     1	0x000000000040667d: addr_gpa2hva at kvm_util.c:1382
     2	 (inlined by) addr_gpa2hva at kvm_util.c:1376
     3	 (inlined by) addr_gva2hva at kvm_util.c:2245
     4	0x0000000000402909: lookup_pfn at access_tracking_perf_test.c:98
     5	 (inlined by) mark_vcpu_memory_idle at access_tracking_perf_test.c:152
     6	 (inlined by) vcpu_thread_main at access_tracking_perf_test.c:232
     7	0x00007fd02d1cb431: ?? ??:0
     8	0x00007fd02d0f9912: ?? ??:0

No vm physical memory at 0xfcbffff000

(and my cpuid hack reducing guest physical address space by half doesn't seem to help either)

Signature · 1 line
Vitaly