KVM: SVM: reduce guest MAXPHYADDR by one in case C-bit is a physical bit
RFC PATCH 1/1 KVM: SVM: reduce guest MAXPHYADDR by one in case C-bit is a physical bit
Several selftests (memslot_modification_stress_test, kvm_page_table_test, dirty_log_perf_test,.. ) which rely on vm_get_max_gfn() started to fail since commit ef4c9f4f65462 ("KVM: selftests: Fix 32-bit truncation of vm_get_max_gfn()") on AMD EPYC 7401P:
./tools/testing/selftests/kvm/demand_paging_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffffbffff000 Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false
pid=47131 tid=47134 errno=0 - Success
1 0x000000000040281b: vcpu_worker at demand_paging_test.c:63
2 0x00007fb36716e431: ?? ??:0
3 0x00007fb36709c912: ?? ??:0Invalid guest sync status: exit_reason=SHUTDOWN
The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:
Address sizes: 43 bits physical, 48 bits virtual
The guest physical address space, however, is not reduced as stated in commit e39f00f60ebd ("KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):
Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.
Implement the reduction.
---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.
--- arch/x86/kvm/cpuid.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
arch/x86/kvm/cpuid.c +10 -3
@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* * If TDP is enabled but an explicit guest MAXPHYADDR is not * provided, use the raw bare metal MAXPHYADDR as reductions to -* the HPAs do not affect GPAs. +* the HPAs do not affect GPAs. The value, however, has to be +* reduced by 1 in case C-bit is a physical bit (APM section +* 15.34.6). */ -if (!tdp_enabled) +if (!tdp_enabled) { g_phys_as = boot_cpu_data.x86_phys_bits; -else if (!g_phys_as) + }else if (!g_phys_as) { g_phys_as = phys_as; +if (kvm_cpu_cap_has (X86_FEATURE_SEV ) && + (cpuid_ebx (0x8000001f ) &0x3f ) < g_phys_as) + g_phys_as -=1 ; + } + entry->eax = g_phys_as | (virt_as <<8 ); entry->edx =0 ;cpuid_entry_override (entry,CPUID_8000_0008_EBX );
Signature · 1 line
2.31.1
Sean Christopherson
maintainerQuoted 8 lines · Vitaly Kuznetsov
Several selftests (
memslot_modification_stress_test,kvm_page_table_test,dirty_log_perf_test,.. ) which rely onvm_get_max_gfn()started to fail since commitef4c9f4f65462("KVM: selftests: Fix 32-bit truncation ofvm_get_max_gfn()") on AMD EPYC 7401P:
./tools/testing/selftests/kvm/demand_paging_testTesting guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset:0xffffbffff000
This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]
Quoted 51 lines
Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success1 0x000000000040281b: vcpu_worker at demand_paging_test.c:63 2 0x00007fb36716e431: ?? ??:0 3 0x00007fb36709c912: ?? ??:0Invalid guest sync status: exit_reason=SHUTDOWN
The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:
Address sizes: 43 bits physical, 48 bits virtual
The guest physical address space, however, is not reduced as stated in commit
e39f00f60ebd("KVM: x86: Use kernel'sx86_phys_bitsto handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.
Implement the reduction.
Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)Signed-off-by Vitaly Kuznetsov <[email protected]>---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.--- arch/x86/kvm/cpuid.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)arch/x86/kvm/cpuid.c +3
-1@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* * If TDP is enabled but an explicit guest MAXPHYADDR is not * provided, use the raw bare metal MAXPHYADDR as reductions to -* the HPAs do not affect GPAs. +* the HPAs do not affect GPAs. The value, however, has to be +* reduced by 1 in case C-bit is a physical bit (APM section +* 15.34.6). */
arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937
Quoted 10 lines
arch/x86/kvm/cpuid.c +5
-2-if (!tdp_enabled) +if (!tdp_enabled) { g_phys_as = boot_cpu_data.x86_phys_bits; -else if (!g_phys_as) + }else if (!g_phys_as) { g_phys_as = phys_as; +if (kvm_cpu_cap_has (X86_FEATURE_SEV ) && + (cpuid_ebx (0x8000001f ) &0x3f ) < g_phys_as) + g_phys_as -=1 ;
This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]
Quoted 8 lines
arch/x86/kvm/cpuid.c +2
-0+ } + entry->eax = g_phys_as | (virt_as <<8 ); entry->edx =0 ;cpuid_entry_override (entry,CPUID_8000_0008_EBX );Signature · 1 line
2.31.1
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov
Several selftests (
memslot_modification_stress_test,kvm_page_table_test,dirty_log_perf_test,.. ) which rely onvm_get_max_gfn()started to fail since commitef4c9f4f65462("KVM: selftests: Fix 32-bit truncation ofvm_get_max_gfn()") on AMD EPYC 7401P:
Quoted 7 lines
Quoted 3 lines
./tools/testing/selftests/kvm/demand_paging_testTesting guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset:0xffffbffff000This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].
Hi Vitaly and everyone!
You are the 3rd person to suffer from this issue :-( Sean Christopherson was first, I was second.
I reported this, then I think we found out that it is not the HyperTransport region after all, and I think that the whole thing lost in 'trying to get answers from AMD'.
https://lore.kernel.org/lkml/[email protected]/
I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.
Sorry that you had to debug this.
Best regards,
Maxim Levitsky
Quoted 80 lines
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]Quoted 51 lines
Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success1 0x000000000040281b: vcpu_worker at demand_paging_test.c:63 2 0x00007fb36716e431: ?? ??:0 3 0x00007fb36709c912: ?? ??:0Invalid guest sync status: exit_reason=SHUTDOWN
The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:
Address sizes: 43 bits physical, 48 bits virtual
The guest physical address space, however, is not reduced as stated in commit
e39f00f60ebd("KVM: x86: Use kernel'sx86_phys_bitsto handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.
Implement the reduction.
Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)Signed-off-by Vitaly Kuznetsov <[email protected]>---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.--- arch/x86/kvm/cpuid.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)arch/x86/kvm/cpuid.c +3
-1@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* * If TDP is enabled but an explicit guest MAXPHYADDR is not * provided, use the raw bare metal MAXPHYADDR as reductions to -* the HPAs do not affect GPAs. +* the HPAs do not affect GPAs. The value, however, has to be +* reduced by 1 in case C-bit is a physical bit (APM section +* 15.34.6). */ arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937
Quoted 10 lines
arch/x86/kvm/cpuid.c +5
-2-if (!tdp_enabled) +if (!tdp_enabled) { g_phys_as = boot_cpu_data.x86_phys_bits; -else if (!g_phys_as) + }else if (!g_phys_as) { g_phys_as = phys_as; +if (kvm_cpu_cap_has (X86_FEATURE_SEV ) && + (cpuid_ebx (0x8000001f ) &0x3f ) < g_phys_as) + g_phys_as -=1 ;This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]
Quoted 8 lines
arch/x86/kvm/cpuid.c +2
-0+ } + entry->eax = g_phys_as | (virt_as <<8 ); entry->edx =0 ;cpuid_entry_override (entry,CPUID_8000_0008_EBX );Signature · 1 line
2.31.1
Quoted 22 lines · Maxim Levitsky
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Vitaly Kuznetsov
Several selftests (
memslot_modification_stress_test,kvm_page_table_test,dirty_log_perf_test,.. ) which rely onvm_get_max_gfn()started to fail since commitef4c9f4f65462("KVM: selftests: Fix 32-bit truncation ofvm_get_max_gfn()") on AMD EPYC 7401P:Quoted 7 lines
Quoted 3 lines
./tools/testing/selftests/kvm/demand_paging_testTesting guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset:0xffffbffff000This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].
Hi Vitaly and everyone!
You are the 3rd person to suffer from this issue :-( Sean Christopherson was first, I was second.
I reported this, then I think we found out that it is not the HyperTransport region after all, and I think that the whole thing lost in 'trying to get answers from AMD'.
Quoted 7 lines
https://lore.kernel.org/lkml/[email protected]/
I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.
Sorry that you had to debug this.
I didn't spend too much time on this, that's the reson for 'RFC' :-) I agree we need at least a short-term solution as permanently failing tests may start masking newly introduces issues.
Signature · 1 line
Vitaly
Paolo Bonzini
maintainerQuoted 2 lines · Maxim Levitsky
I'll say, a hack to reduce it by 1 bit is still better that failing tests, at least until AMD explains to us, about what is going on.
What's going on is documented in the thread at
https://yhbt.net/lore/all/[email protected]/:
Quoted 9 lines
That doesn't really follow what Andrew gave us, namely:
1) On parts with <40 bits, its fully hidden from software
2) Before Fam17h, it was always 12G just below 1T, even if there was more RAM above this location
3) On Fam17h and later, it is variable based on SME, and is either just below 2^48 (no encryption) or 2^43 (encryption)
If you can use this information to implement the fix, that'd be very nice. I didn't apply the hackish fix because I wanted to test it on a SME-enabled box.
Paolo
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Vitaly Kuznetsov
Several selftests (
memslot_modification_stress_test,kvm_page_table_test,dirty_log_perf_test,.. ) which rely onvm_get_max_gfn()started to fail since commitef4c9f4f65462("KVM: selftests: Fix 32-bit truncation ofvm_get_max_gfn()") on AMD EPYC 7401P:
./tools/testing/selftests/kvm/demand_paging_testTesting guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset:0xffffbffff000
Quoted 5 lines
This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].
Hey,
it seems I'm only three months late to the party!
Quoted 3 lines
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]
This patch helps indeed, thanks! Paolo, any particular reason you haven't queued it yet?
Quoted 58 lines
Quoted 58 lines
Finished creating vCPUs and starting uffd threads Started all vCPUs
==== Test Assertion Failure ====
demand_paging_test.c:63: false pid=47131 tid=47134 errno=0 - Success1 0x000000000040281b: vcpu_worker at demand_paging_test.c:63 2 0x00007fb36716e431: ?? ??:0 3 0x00007fb36709c912: ?? ??:0Invalid guest sync status: exit_reason=SHUTDOWN
The commit, however, seems to be correct, it just revealed an already present issue. AMD CPUs which support SEV may have a reduced physical address space, e.g. on AMD EPYC 7401P I see:
Address sizes: 43 bits physical, 48 bits virtual
The guest physical address space, however, is not reduced as stated in commit
e39f00f60ebd("KVM: x86: Use kernel'sx86_phys_bitsto handle reduced MAXPHYADDR"). This seems to be almost correct, however, APM has one more clause (15.34.6):Note that because guest physical addresses are always translated through the nested page tables, the size of the guest physical address space is not impacted by any physical address space reduction indicated in CPUID 8000_001F[EBX]. If the C-bit is a physical address bit however, the guest physical address space is effectively reduced by 1 bit.
Implement the reduction.
Fixes e39f00f60ebd (KVM: x86: Use kernel's x86_phys_bits to handle reduced MAXPHYADDR)Signed-off-by Vitaly Kuznetsov <[email protected]>---
- RFC: I may have misdiagnosed the problem as I didn't dig to where exactly the guest crashes.--- arch/x86/kvm/cpuid.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)arch/x86/kvm/cpuid.c +5
-3@@ -923,13 +923,20 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* * If TDP is enabled but an explicit guest MAXPHYADDR is not * provided, use the raw bare metal MAXPHYADDR as reductions to -* the HPAs do not affect GPAs. +* the HPAs do not affect GPAs. The value, however, has to be +* reduced by 1 in case C-bit is a physical bit (APM section +* 15.34.6). */ -if (!tdp_enabled) +if (!tdp_enabled) { g_phys_as = boot_cpu_data.x86_phys_bits; -else if (!g_phys_as) + }else if (!g_phys_as) { g_phys_as = phys_as;
Quoted 9 lines
arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 937
Quoted 3 lines
arch/x86/kvm/cpuid.c +3
-0+if (kvm_cpu_cap_has (X86_FEATURE_SEV ) && + (cpuid_ebx (0x8000001f ) &0x3f ) < g_phys_as) + g_phys_as -=1 ;This is incorrect, non-SEV guests do not see a reduced address space. See Tom's explanation[*]
I see, thanks for the pointer.
Quoted 9 lines
Quoted 8 lines
arch/x86/kvm/cpuid.c +2
-0+ } + entry->eax = g_phys_as | (virt_as <<8 ); entry->edx =0 ;cpuid_entry_override (entry,CPUID_8000_0008_EBX );Signature · 1 line
2.31.1
Signature · 1 line
Vitaly
Quoted 20 lines · Vitaly Kuznetsov
Quoted 10 lines · Sean Christopherson
Quoted 8 lines · Vitaly Kuznetsov
Several selftests (
memslot_modification_stress_test,kvm_page_table_test,dirty_log_perf_test,.. ) which rely onvm_get_max_gfn()started to fail since commitef4c9f4f65462("KVM: selftests: Fix 32-bit truncation ofvm_get_max_gfn()") on AMD EPYC 7401P:
./tools/testing/selftests/kvm/demand_paging_testTesting guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset:0xffffbffff000Quoted 5 lines
This look a lot like the signature I remember from the original bug[1]. I assume you're hitting the magic HyperTransport region[2]. I thought that was fixed, but the hack-a-fix for selftests never got applied[3].
Hey,
Quoted 7 lines
it seems I'm only three months late to the party!
Quoted 3 lines
[2] https://lkml.kernel.org/r/[email protected]
[3] https://lkml.kernel.org/r/[email protected]This patch helps indeed
FWIW, 'access_tracking_perf_test' remains broken even after the patch is applied:
# ./access_tracking_perf_test Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xfffcbffff000
Populating memory : 3.858448918s
Writing to populated memory : 0.937319626s
Reading from populated memory : 0.003073207s ==== Test Assertion Failure ====
lib/kvm_util.c:1382: false
pid=6422 tid=6425 errno=4 - Interrupted system call
1 0x000000000040667d: addr_gpa2hva at kvm_util.c:1382
2 (inlined by) addr_gpa2hva at kvm_util.c:1376
3 (inlined by) addr_gva2hva at kvm_util.c:2245
4 0x0000000000402909: lookup_pfn at access_tracking_perf_test.c:98
5 (inlined by) mark_vcpu_memory_idle at access_tracking_perf_test.c:152
6 (inlined by) vcpu_thread_main at access_tracking_perf_test.c:232
7 0x00007fd02d1cb431: ?? ??:0
8 0x00007fd02d0f9912: ?? ??:0 No vm physical memory at 0xfcbffff000
(and my cpuid hack reducing guest physical address space by half doesn't seem to help either)
Signature · 1 line
Vitaly