Skip to contentSkip to conversation

x86/access: Fixed test stuck issue on new 52bit machine

PATCH In review Yang Weijiang 2 participants 5 messages Jan 10, 2021 – Jan 13, 2021 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/access: Fixed test stuck issue on new 52bit machine

When the application is tested on a machine with 52bit-physical-address, the synthesized 52bit GPA triggers EPT(4-Level) fast_page_fault infinitely. On the other hand, there's no reserved bits in 51:max_physical_address on machines with 52bit-physical-address.

---
 x86/access.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
x86/access.c +11 -9
index 7dc9eb6..bec1c4d 100644
@@ -15,6 +15,7 @@ static _Bool verbose = false;
 typedef unsigned long pt_element_t;
 static int invalid_mask;
 static int page_table_levels;
+static int max_phyaddr;
 
 #define PT_BASE_ADDR_MASK ((pt_element_t)((((pt_element_t)1 << 36) - 1) & PAGE_MASK))
 #define PT_PSE_BASE_ADDR_MASK (PT_BASE_ADDR_MASK & ~(1ull << 21))
@@ -394,9 +395,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
     if (!F(AC_PDE_ACCESSED))
         at->ignore_pde = PT_ACCESSED_MASK;
 
-    pde_valid = F(AC_PDE_PRESENT)
-        && !F(AC_PDE_BIT51) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
+    pde_valid = F(AC_PDE_PRESENT) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
         && !(F(AC_PDE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pde_valid &= !F(AC_PDE_BIT51);
 
     if (!pde_valid) {
         at->expected_fault = 1;
@@ -420,9 +422,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
 
     at->expected_pde |= PT_ACCESSED_MASK;
 
-    pte_valid = F(AC_PTE_PRESENT)
-        && !F(AC_PTE_BIT51) && !F(AC_PTE_BIT36)
+    pte_valid = F(AC_PTE_PRESENT) && !F(AC_PTE_BIT36)
         && !(F(AC_PTE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pte_valid &= !F(AC_PTE_BIT51);
 
     if (!pte_valid) {
         at->expected_fault = 1;
@@ -964,13 +967,11 @@ static int ac_test_run(void)
     shadow_cr4 = read_cr4();
     shadow_efer = rdmsr(MSR_EFER);
 
-    if (cpuid_maxphyaddr() >= 52) {
-        invalid_mask |= AC_PDE_BIT51_MASK;
-        invalid_mask |= AC_PTE_BIT51_MASK;
-    }
-    if (cpuid_maxphyaddr() >= 37) {
+    if (max_phyaddr  >= 37 && max_phyaddr < 52) {
         invalid_mask |= AC_PDE_BIT36_MASK;
         invalid_mask |= AC_PTE_BIT36_MASK;
+        invalid_mask |= AC_PDE_BIT51_MASK;
+        invalid_mask |= AC_PTE_BIT51_MASK;
     }
 
     if (this_cpu_has(X86_FEATURE_PKU)) {
@@ -1038,6 +1039,7 @@ int main(void)
     int r;
 
     printf("starting test\n\n");
+    max_phyaddr = cpuid_maxphyaddr();
     page_table_levels = 4;
     r = ac_test_run();
 
Signature · 1 line
2.17.2
#1 lore raw
Quoted 2 lines · Yang Weijiang

When the application is tested on a machine with 52bit-physical-address, the synthesized 52bit GPA triggers EPT(4-Level) fast_page_fault infinitely.

That doesn't sound right, KVM should use 5-level EPT if guest maxpa > 48. Hmm, unless the CPU doesn't support 5-level EPT, but I didn't think such CPUs (maxpa=52 w/o 5-level EPT) existed? Ah, but it would be possible with nested VMX, and initial KVM 5-level support didn't allow nested 5-level EPT. Any chance you're running this test in a VM with 5-level EPT disabled, but maxpa=52?

Quoted 34 lines

On the other hand, there's no reserved bits in 51:max_physical_address on machines with 52bit-physical-address.

---
 x86/access.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
x86/access.c +4 -2
index 7dc9eb6..bec1c4d 100644
@@ -15,6 +15,7 @@ static _Bool verbose = false;
 typedef unsigned long pt_element_t;
 static int invalid_mask;
 static int page_table_levels;
+static int max_phyaddr;
 
 #define PT_BASE_ADDR_MASK ((pt_element_t)((((pt_element_t)1 << 36) - 1) & PAGE_MASK))
 #define PT_PSE_BASE_ADDR_MASK (PT_BASE_ADDR_MASK & ~(1ull << 21))
@@ -394,9 +395,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
     if (!F(AC_PDE_ACCESSED))
         at->ignore_pde = PT_ACCESSED_MASK;
 
-    pde_valid = F(AC_PDE_PRESENT)
-        && !F(AC_PDE_BIT51) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
+    pde_valid = F(AC_PDE_PRESENT) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
         && !(F(AC_PDE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pde_valid &= !F(AC_PDE_BIT51);
 
     if (!pde_valid) {
         at->expected_fault = 1;

x86/access.c static void ac_emulate_access(ac_test_t *at, unsigned flags) · line 428

Quoted 9 lines
x86/access.c +3 -2
@@ -420,9 +422,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
 
     at->expected_pde |= PT_ACCESSED_MASK;
 
-    pte_valid = F(AC_PTE_PRESENT)
-        && !F(AC_PTE_BIT51) && !F(AC_PTE_BIT36)
+    pte_valid = F(AC_PTE_PRESENT) && !F(AC_PTE_BIT36)
         && !(F(AC_PTE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pte_valid &= !F(AC_PTE_BIT51);

This _should_ be unnecessary. As below, AC_*_BIT51_MASK will be set in invalid_mask, and so ac_test_bump_one() will skip tests that try to set bit 51.

Quoted 7 lines
x86/access.c +0 -1
     if (!pte_valid) {
         at->expected_fault = 1;
@@ -964,13 +967,11 @@ static int ac_test_run(void)
     shadow_cr4 = read_cr4();
     shadow_efer = rdmsr(MSR_EFER);
 
-    if (cpuid_maxphyaddr() >= 52) {

x86/access.c static int ac_test_run(void) · line 975

Quoted 10 lines
x86/access.c +3 -4
-        invalid_mask |= AC_PDE_BIT51_MASK;
-        invalid_mask |= AC_PTE_BIT51_MASK;
-    }
-    if (cpuid_maxphyaddr() >= 37) {
+    if (max_phyaddr  >= 37 && max_phyaddr < 52) {
         invalid_mask |= AC_PDE_BIT36_MASK;
         invalid_mask |= AC_PTE_BIT36_MASK;
+        invalid_mask |= AC_PDE_BIT51_MASK;
+        invalid_mask |= AC_PTE_BIT51_MASK;
     }

This change is incorrect. "invalid_mask" is misleading in this context as it means "bits that can't be tested because they're legal". So setting the bit 51 flags in invalid_mask if 'maxpa >= 52' is correct, as it states those tests are "invalid" because setting bit 51 will not fault.

All that being said, it's also entirely possible I'm misreading this test, I've done it many times before :-)

Quoted 12 lines
x86/access.c +1 -0
     if (this_cpu_has(X86_FEATURE_PKU)) {
@@ -1038,6 +1039,7 @@ int main(void)
     int r;
 
     printf("starting test\n\n");
+    max_phyaddr = cpuid_maxphyaddr();
     page_table_levels = 4;
     r = ac_test_run();
 
Signature · 1 line
2.17.2
#2 lore raw
Quoted 10 lines · Sean Christopherson
Quoted 2 lines · Yang Weijiang

When the application is tested on a machine with 52bit-physical-address, the synthesized 52bit GPA triggers EPT(4-Level) fast_page_fault infinitely.

That doesn't sound right, KVM should use 5-level EPT if guest maxpa > 48. Hmm, unless the CPU doesn't support 5-level EPT, but I didn't think such CPUs (maxpa=52 w/o 5-level EPT) existed? Ah, but it would be possible with nested VMX, and initial KVM 5-level support didn't allow nested 5-level EPT. Any chance you're running this test in a VM with 5-level EPT disabled, but maxpa=52?

Hi, Sean,
Thanks for the reply!
I use default settings of the unit-test + 5.2.0 QEMU + 5.10 kernel, in this case, QEMU uses cpu->phys_bits==40, so the guest's PA=40bit and LA=57bit, hence 5-level EPT is not enabled. My physical machine is PA=52 and LA=57 as can checked from cpuid:
cpuid -1r -l 0x80000008 -s 0
CPU:
0x80000008 0x00: eax=0x00003934 ...
There're two other ways to w/a this issue: 1) change the QEMU params to to extra_params = -cpu host,host-phys-bits, so guest's PA=52 and LA=57, this will enable 5-level EPT, meanwhile, it escapes the problematic GPA by adding AC_*_BIT51_MASK in invalid_mask.

2) add allow_smaller_maxphyaddr=1 to kvm-intel module.

the perf trace looks like this:
12481.879 qemu-system-x8/27004 kvm:kvm_page_fault:address 8000002000000 error_code 181

Quoted 37 lines
Quoted 37 lines

On the other hand, there's no reserved bits in 51:max_physical_address on machines with 52bit-physical-address.

---
 x86/access.c | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
x86/access.c +4 -2
index 7dc9eb6..bec1c4d 100644
@@ -15,6 +15,7 @@ static _Bool verbose = false;
 typedef unsigned long pt_element_t;
 static int invalid_mask;
 static int page_table_levels;
+static int max_phyaddr;
 
 #define PT_BASE_ADDR_MASK ((pt_element_t)((((pt_element_t)1 << 36) - 1) & PAGE_MASK))
 #define PT_PSE_BASE_ADDR_MASK (PT_BASE_ADDR_MASK & ~(1ull << 21))
@@ -394,9 +395,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
     if (!F(AC_PDE_ACCESSED))
         at->ignore_pde = PT_ACCESSED_MASK;
 
-    pde_valid = F(AC_PDE_PRESENT)
-        && !F(AC_PDE_BIT51) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
+    pde_valid = F(AC_PDE_PRESENT) && !F(AC_PDE_BIT36) && !F(AC_PDE_BIT13)
         && !(F(AC_PDE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pde_valid &= !F(AC_PDE_BIT51);
 
     if (!pde_valid) {
         at->expected_fault = 1;
@@ -420,9 +422,10 @@ static void ac_emulate_access(ac_test_t *at, unsigned flags)
 
     at->expected_pde |= PT_ACCESSED_MASK;
 
Quoted 10 lines

x86/access.c static void ac_emulate_access(ac_test_t *at, unsigned flags) · line 428

Quoted 6 lines
x86/access.c +3 -2
-    pte_valid = F(AC_PTE_PRESENT)
-        && !F(AC_PTE_BIT51) && !F(AC_PTE_BIT36)
+    pte_valid = F(AC_PTE_PRESENT) && !F(AC_PTE_BIT36)
         && !(F(AC_PTE_NX) && !F(AC_CPU_EFER_NX));
+    if (max_phyaddr < 52)
+        pte_valid &= !F(AC_PTE_BIT51);

This _should_ be unnecessary. As below, AC_*_BIT51_MASK will be set in invalid_mask, and so ac_test_bump_one() will skip tests that try to set bit 51.

These code is to avoid some "unexpected access" messages on some platforms if below change is added.

Quoted 6 lines
Quoted 6 lines
x86/access.c +0 -0
     if (!pte_valid) {
         at->expected_fault = 1;
@@ -964,13 +967,11 @@ static int ac_test_run(void)
     shadow_cr4 = read_cr4();
     shadow_efer = rdmsr(MSR_EFER);
 
Quoted 16 lines

x86/access.c static int ac_test_run(void) · line 975

Quoted 11 lines
x86/access.c +3 -5
-    if (cpuid_maxphyaddr() >= 52) {
-        invalid_mask |= AC_PDE_BIT51_MASK;
-        invalid_mask |= AC_PTE_BIT51_MASK;
-    }
-    if (cpuid_maxphyaddr() >= 37) {
+    if (max_phyaddr  >= 37 && max_phyaddr < 52) {
         invalid_mask |= AC_PDE_BIT36_MASK;
         invalid_mask |= AC_PTE_BIT36_MASK;
+        invalid_mask |= AC_PDE_BIT51_MASK;
+        invalid_mask |= AC_PTE_BIT51_MASK;
     }

This change is incorrect. "invalid_mask" is misleading in this context as it means "bits that can't be tested because they're legal". So setting the bit 51 flags in invalid_mask if 'maxpa >= 52' is correct, as it states those tests are "invalid" because setting bit 51 will not fault.

Maybe I misunderstood the purpose of this test, so I skipped the non-fault case, e.g., when maxpa >=52. In guest PA=40 case, AC_*_BIT51_MASK bits are not added here, then when they're set in guest page_table entries, does it expect a fault? What's the expected result on a real 52bit platform?

Quoted 3 lines

All that being said, it's also entirely possible I'm misreading this test, I've done it many times before :-)

I found it's hard to pass all the enclosed tests on various platforms :-)

Quoted 12 lines
Quoted 12 lines
x86/access.c +1 -0
     if (this_cpu_has(X86_FEATURE_PKU)) {
@@ -1038,6 +1039,7 @@ int main(void)
     int r;
 
     printf("starting test\n\n");
+    max_phyaddr = cpuid_maxphyaddr();
     page_table_levels = 4;
     r = ac_test_run();
 
Signature · 1 line
2.17.2
#3 lore raw
Quoted 11 lines · Yang Weijiang
Quoted 10 lines · Sean Christopherson
Quoted 2 lines · Yang Weijiang

When the application is tested on a machine with 52bit-physical-address, the synthesized 52bit GPA triggers EPT(4-Level) fast_page_fault infinitely.

That doesn't sound right, KVM should use 5-level EPT if guest maxpa > 48. Hmm, unless the CPU doesn't support 5-level EPT, but I didn't think such CPUs (maxpa=52 w/o 5-level EPT) existed? Ah, but it would be possible with nested VMX, and initial KVM 5-level support didn't allow nested 5-level EPT. Any chance you're running this test in a VM with 5-level EPT disabled, but maxpa=52?

Quoted 3 lines

Hi, Sean,
Thanks for the reply!
I use default settings of the unit-test + 5.2.0 QEMU + 5.10 kernel, in

The default settings are supposed to set guest.MAXPA = host.MAXPA. At least, I assume that's the purpose of '-cpu max'. Maybe your copy of kvm-unit-tests' x86/unittests.cfg is stale?

[access]
file = access.flat
arch = x86_64
extra_params = -cpu max
timeout = 180

Quoted 5 lines

this case, QEMU uses cpu->phys_bits==40, so the guest's PA=40bit and LA=57bit, hence 5-level EPT is not enabled. My physical machine is PA=52 and LA=57 as can checked from cpuid:
cpuid -1r -l 0x80000008 -s 0
CPU:

Quoted 7 lines
   0x80000008 0x00: eax=0x00003934 ...

There're two other ways to w/a this issue: 1) change the QEMU params to to extra_params = -cpu host,host-phys-bits, so guest's PA=52 and LA=57, this will enable 5-level EPT, meanwhile, it escapes the problematic GPA by adding AC_*_BIT51_MASK in invalid_mask.

2) add allow_smaller_maxphyaddr=1 to kvm-intel module.

Setting allow_smaller_maxphyaddr=1 is the correct answer.

#4 lore raw
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Yang Weijiang
Quoted 10 lines · Sean Christopherson
Quoted 2 lines · Yang Weijiang

When the application is tested on a machine with 52bit-physical-address, the synthesized 52bit GPA triggers EPT(4-Level) fast_page_fault infinitely.

That doesn't sound right, KVM should use 5-level EPT if guest maxpa > 48. Hmm, unless the CPU doesn't support 5-level EPT, but I didn't think such CPUs (maxpa=52 w/o 5-level EPT) existed? Ah, but it would be possible with nested VMX, and initial KVM 5-level support didn't allow nested 5-level EPT. Any chance you're running this test in a VM with 5-level EPT disabled, but maxpa=52?

Quoted 3 lines

Hi, Sean,
Thanks for the reply!
I use default settings of the unit-test + 5.2.0 QEMU + 5.10 kernel, in

The default settings are supposed to set guest.MAXPA = host.MAXPA. At least, I assume that's the purpose of '-cpu max'. Maybe your copy of kvm-unit-tests' x86/unittests.cfg is stale?

Quoted 6 lines

[access]
file = access.flat
arch = x86_64
extra_params = -cpu max
timeout = 180

Yes, I used the default max option, but looks like guest max physical address is forgotten somehow in this case. Anyway, I dropped a patch to QEMU community to enable it.

Quoted 14 lines
Quoted 5 lines

this case, QEMU uses cpu->phys_bits==40, so the guest's PA=40bit and LA=57bit, hence 5-level EPT is not enabled. My physical machine is PA=52 and LA=57 as can checked from cpuid:
cpuid -1r -l 0x80000008 -s 0
CPU:

Quoted 7 lines
   0x80000008 0x00: eax=0x00003934 ...

There're two other ways to w/a this issue: 1) change the QEMU params to to extra_params = -cpu host,host-phys-bits, so guest's PA=52 and LA=57, this will enable 5-level EPT, meanwhile, it escapes the problematic GPA by adding AC_*_BIT51_MASK in invalid_mask.

2) add allow_smaller_maxphyaddr=1 to kvm-intel module.

Setting allow_smaller_maxphyaddr=1 is the correct answer.