KVM: X86: Check pte present first in __shadow_walk_next()
PATCH 1/2 KVM: X86: Check pte present first in __shadow_walk_next()
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the pte is present before calling __shadow_walk_next(). But checking pte present in __shadow_walk_next() is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing is_shadow_present_pte() in the loop bodies.
--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -2231,7 +2231,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void __shadow_walk_next (struct kvm_shadow_walk_iterator *iterator,u64 spte) { -if (is_last_spte (spte, iterator->level)) { +if (!is_shadow_present_pte (spte) ||is_last_spte (spte, iterator->level)) { iterator->level =0 ;return ; }
Signature · 1 line
2.19.1.6.gb485710b
PATCH 2/2 KVM: X86: Remove the present check from for_each_shadow_entry* loop body
From: Lai Jiangshan <[email protected]>
The function __shadow_walk_next() for the for_each_shadow_entry* looping has the check and propagates the result to shadow_walk_okay().
--- arch/x86/kvm/mmu/mmu.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -10
@@ -3152,9 +3152,6 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte)for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { sptep = iterator.sptep; *spte = old_spte; - -if (!is_shadow_present_pte (old_spte)) -break ; }return sptep;
@@ -3694,9 +3691,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level spte =mmu_spte_get_lockless (iterator.sptep); sptes[leaf] = spte; - -if (!is_shadow_present_pte (spte)) -break ; }return leaf;
@@ -3811,11 +3805,8 @@ static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)u64 spte;walk_shadow_page_lockless_begin (vcpu); -for_each_shadow_entry_lockless (vcpu, addr, iterator, spte) { +for_each_shadow_entry_lockless (vcpu, addr, iterator, spte)clear_sp_write_flooding_count (iterator.sptep); -if (!is_shadow_present_pte (spte)) -break ; - }walk_shadow_page_lockless_end (vcpu); }
Signature · 1 line
2.19.1.6.gb485710b
Sean Christopherson
maintainerQuoted 7 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the pte is present before calling
__shadow_walk_next(). But checking pte present in__shadow_walk_next()is a more prudent way of programing and loop bodies will not need to always check it. It allows us removingis_shadow_present_pte()in the loop bodies.
There needs to be more analysis in the changelog, as there are many more callers to __shadow_walk_next() than the three that are modified in the next patch. It might even make sense to squash the two patches together, i.e. make it a "move" instead of an "add + remove", and then explicitly explain why it's ok to add the check for paths that do _not_ currently have a !is_shadow_present_pte() in the loop body.
Specifically, FNAME(fetch) via shadow_walk_next() and __direct_map() via for_each_shadow_entry() do not currently terminate their walks with a !PRESENT, but they get away with it because they install present non-leaf SPTEs in the loop itself.
The other argument for the audit (changelog+patch) of all users is that the next patch misses FNAME(invlpg), e.g.
@@ -977,7 +980,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)
FNAME(update_pte)(vcpu, sp, sptep, &gpte);
}- if (!is_shadow_present_pte(*sptep) || !sp->unsync_children)
+ if (!sp->unsync_children)
break;
}
write_unlock(&vcpu->kvm->mmu_lock);It would also be worthwhile to document via the changelog that terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage iter->shadow_addr.
And for clarity and safety, I think it would be worth adding the patch below as a prep patch to document and enforce that walking the non-leaf SPTEs when faulting in a page should never terminate early.
From 1c202a7e82b1931e4eb37b23aa9d7108340a6cd2 Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Thu, 12 Aug 2021 08:38:35 -0700
Subject: [PATCH] KVM: x86/mmu: Verify shadow walk doesn't terminate early in
page faults
WARN and bail if the shadow walk for faulting in a SPTE terminates early, i.e. doesn't reach the expected level because the walk encountered a terminal SPTE. The shadow walks for page faults are subtle in that they install non-leaf SPTEs (zapping leaf SPTEs if necessary!) in the loop body, and consume the newly created non-leaf SPTE in the loop control, e.g. __shadow_walk_next(). In other words, the walks guarantee that the walk will stop if and only if the target level is reached by installing non-leaf SPTEs to guarantee the walk remains valid.
Opportunistically use fault->goal-level instead of it.level in FNAME(fetch) to further clarify that KVM always installs the leaf SPTE at the target level.
--- arch/x86/kvm/mmu/mmu.c | 3 +++ arch/x86/kvm/mmu/paging_tmpl.h | 7 +++++-- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -2992,6 +2992,9 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)account_huge_nx_page (vcpu->kvm, sp); } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep,ACC_ALL , fault->write, fault->goal_level, base_gfn, fault->pfn, fault->prefault, fault->map_writable);
arch/x86/kvm/mmu/paging_tmpl.h +5 -2
@@ -749,9 +749,12 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, } } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep, gw->pte_access, fault->write, - it.level, base_gfn, fault->pfn, fault->prefault, - fault->map_writable); + fault->goal_level, base_gfn, fault->pfn, + fault->prefault, fault->map_writable);if (ret ==RET_PF_SPURIOUS )return ret;
--
2.33.0.rc1.237.g0d66db33f3-goog
PATCH v2 1/1 KVM: X86: Move PTE present check from loop body to __shadow_walk_next()
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the PTE is present before calling __shadow_walk_next(): Some loop bodies simply exit with a !PRESENT directly and some other loop bodies, i.e. FNAME(fetch) and __direct_map() do not currently terminate their walks with a !PRESENT, but they get away with it because they install present non-leaf SPTEs in the loop itself.
But checking pte present in __shadow_walk_next() is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing unneded is_shadow_present_pte() in the loop bodies.
Terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage iter->shadow_addr. Even some paths that do _not_ currently have a !is_shadow_present_pte() in the loop body is Ok since they will install present non-leaf SPTEs and the additinal present check is just an NOP.
The checking result in __shadow_walk_next() will be propagated to shadow_walk_okay() for being used in any for(;;) loop.
---
Changed from V1:
Merge the two patches
Update changelog
Remove !is_shadow_present_pte() in FNAME(invlpg)
arch/x86/kvm/mmu/mmu.c | 13 ++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 3 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -11
@@ -2231,7 +2231,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void __shadow_walk_next (struct kvm_shadow_walk_iterator *iterator,u64 spte) { -if (is_last_spte (spte, iterator->level)) { +if (!is_shadow_present_pte (spte) ||is_last_spte (spte, iterator->level)) { iterator->level =0 ;return ; }
@@ -3152,9 +3152,6 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte)for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { sptep = iterator.sptep; *spte = old_spte; - -if (!is_shadow_present_pte (old_spte)) -break ; }return sptep;
@@ -3694,9 +3691,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level spte =mmu_spte_get_lockless (iterator.sptep); sptes[leaf] = spte; - -if (!is_shadow_present_pte (spte)) -break ; }return leaf;
@@ -3811,11 +3805,8 @@ static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)u64 spte;walk_shadow_page_lockless_begin (vcpu); -for_each_shadow_entry_lockless (vcpu, addr, iterator, spte) { +for_each_shadow_entry_lockless (vcpu, addr, iterator, spte)clear_sp_write_flooding_count (iterator.sptep); -if (!is_shadow_present_pte (spte)) -break ; - }walk_shadow_page_lockless_end (vcpu); }
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -977,7 +977,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)FNAME (update_pte)(vcpu, sp, sptep, &gpte); } -if (!is_shadow_present_pte (*sptep) || !sp->unsync_children) +if (!sp->unsync_children)break ; }write_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
2.19.1.6.gb485710b
Re: [PATCH V2] KVM: X86: Move PTE present check from loop body to __shadow_walk_next()
Hello, Paolo
Could you have a review please.
Thanks
Lai
Quoted 95 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the PTE is present before calling
__shadow_walk_next(): Some loop bodies simply exit with a !PRESENT directly and some other loop bodies, i.e. FNAME(fetch) and__direct_map()do not currently terminate their walks with a !PRESENT, but they get away with it because they install present non-leaf SPTEs in the loop itself.But checking pte present in
__shadow_walk_next()is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing unnededis_shadow_present_pte()in the loop bodies.Terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage
iter->shadow_addr. Even some paths that do _not_ currently have a !is_shadow_present_pte() in the loop body is Ok since they will install present non-leaf SPTEs and the additinal present check is just an NOP.The checking result in
__shadow_walk_next()will be propagated toshadow_walk_okay()for being used in any for(;;) loop.Signed-off-by Lai Jiangshan <[email protected]>---
Changed from V1:
Merge the two patches
Update changelog
Remove !is_shadow_present_pte() in FNAME(invlpg)arch/x86/kvm/mmu/mmu.c | 13 ++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 3 insertions(+), 12 deletions(-)arch/x86/kvm/mmu/mmu.c +2
-11@@ -2231,7 +2231,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void __shadow_walk_next (struct kvm_shadow_walk_iterator *iterator,u64 spte) { -if (is_last_spte (spte, iterator->level)) { +if (!is_shadow_present_pte (spte) ||is_last_spte (spte, iterator->level)) { iterator->level =0 ;return ; }@@ -3152,9 +3152,6 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte)for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { sptep = iterator.sptep; *spte = old_spte; - -if (!is_shadow_present_pte (old_spte)) -break ; }return sptep;@@ -3694,9 +3691,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level spte =mmu_spte_get_lockless (iterator.sptep); sptes[leaf] = spte; - -if (!is_shadow_present_pte (spte)) -break ; }return leaf;@@ -3811,11 +3805,8 @@ static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)u64 spte;walk_shadow_page_lockless_begin (vcpu); -for_each_shadow_entry_lockless (vcpu, addr, iterator, spte) { +for_each_shadow_entry_lockless (vcpu, addr, iterator, spte)clear_sp_write_flooding_count (iterator.sptep); -if (!is_shadow_present_pte (spte)) -break ; - }walk_shadow_page_lockless_end (vcpu); }arch/x86/kvm/mmu/paging_tmpl.h +1
-2@@ -977,7 +977,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)FNAME (update_pte)(vcpu, sp, sptep, &gpte); } -if (!is_shadow_present_pte (*sptep) || !sp->unsync_children) +if (!sp->unsync_children)break ; }write_unlock (&vcpu->kvm->mmu_lock); --2.19.1.6.gb485710b
Re: [PATCH V2] KVM: X86: Move PTE present check from loop body to __shadow_walk_next()
In reply to Lai JiangshanQuoted 11 lines · Lai Jiangshan
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the PTE is present before calling
__shadow_walk_next(): Some loop bodies simply exit with a !PRESENT directly and some other loop bodies, i.e. FNAME(fetch) and__direct_map()do not currently terminate their walks with a !PRESENT, but they get away with it because they install present non-leaf SPTEs in the loop itself.But checking pte present in
__shadow_walk_next()is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing unnededis_shadow_present_pte()in the loop bodies.
^^^^^^^ unneeded
Quoted 6 lines
Terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage
iter->shadow_addr. Even some paths that do _not_ currently have a !is_shadow_present_pte() in the loop body is Ok since they will install present non-leaf SPTEs and the additinal present check
^^^^^^^^^ additional
Quoted 7 lines
is just an NOP.
The checking result in
__shadow_walk_next()will be propagated toshadow_walk_okay()for being used in any for(;;) loop.Signed-off-by Lai Jiangshan <[email protected]>---
Nits aside,
PATCH v3 1/2 KVM: x86/mmu: Verify shadow walk doesn't terminate early in page faults
From: Sean Christopherson <[email protected]>
WARN and bail if the shadow walk for faulting in a SPTE terminates early, i.e. doesn't reach the expected level because the walk encountered a terminal SPTE. The shadow walks for page faults are subtle in that they install non-leaf SPTEs (zapping leaf SPTEs if necessary!) in the loop body, and consume the newly created non-leaf SPTE in the loop control, e.g. __shadow_walk_next(). In other words, the walks guarantee that the walk will stop if and only if the target level is reached by installing non-leaf SPTEs to guarantee the walk remains valid.
Opportunistically use fault->goal-level instead of it.level in FNAME(fetch) to further clarify that KVM always installs the leaf SPTE at the target level.
--- arch/x86/kvm/mmu/mmu.c | 3 +++ arch/x86/kvm/mmu/paging_tmpl.h | 7 +++++-- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -3006,6 +3006,9 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)account_huge_nx_page (vcpu->kvm, sp); } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep,ACC_ALL , fault->write, fault->goal_level, base_gfn, fault->pfn, fault->prefault, fault->map_writable);
arch/x86/kvm/mmu/paging_tmpl.h +5 -2
@@ -749,9 +749,12 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, } } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep, gw->pte_access, fault->write, - it.level, base_gfn, fault->pfn, fault->prefault, - fault->map_writable); + fault->goal_level, base_gfn, fault->pfn, + fault->prefault, fault->map_writable);if (ret ==RET_PF_SPURIOUS )return ret;
Signature · 1 line
2.19.1.6.gb485710b
PATCH v3 2/2 KVM: X86: Move PTE present check from loop body to __shadow_walk_next()
From: Lai Jiangshan <[email protected]>
So far, the loop bodies already ensure the PTE is present before calling __shadow_walk_next(): Some loop bodies simply exit with a !PRESENT directly and some other loop bodies, i.e. FNAME(fetch) and __direct_map() do not currently terminate their walks with a !PRESENT, but they get away with it because they install present non-leaf SPTEs in the loop itself.
But checking pte present in __shadow_walk_next() is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing unneeded is_shadow_present_pte() in the loop bodies.
Terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage iter->shadow_addr. Even some paths that do _not_ currently have a !is_shadow_present_pte() in the loop body is Ok since they will install present non-leaf SPTEs and the additional present check is just an NOP.
The checking result in __shadow_walk_next() will be propagated to shadow_walk_okay() for being used in any for(;;) loop.
---
Changed from V2:
Fix typo in the changelog reported by Sean Add Reviewed-by from Sean
Changed from V1:
Merge the two patches
Update changelog
Remove !is_shadow_present_pte() in FNAME(invlpg)
arch/x86/kvm/mmu/mmu.c | 13 ++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 3 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -11
@@ -2223,7 +2223,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void __shadow_walk_next (struct kvm_shadow_walk_iterator *iterator,u64 spte) { -if (is_last_spte (spte, iterator->level)) { +if (!is_shadow_present_pte (spte) ||is_last_spte (spte, iterator->level)) { iterator->level =0 ;return ; }
@@ -3159,9 +3159,6 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte)for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { sptep = iterator.sptep; *spte = old_spte; - -if (!is_shadow_present_pte (old_spte)) -break ; }return sptep;
@@ -3721,9 +3718,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level spte =mmu_spte_get_lockless (iterator.sptep); sptes[leaf] = spte; - -if (!is_shadow_present_pte (spte)) -break ; }return leaf;
@@ -3838,11 +3832,8 @@ static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)u64 spte;walk_shadow_page_lockless_begin (vcpu); -for_each_shadow_entry_lockless (vcpu, addr, iterator, spte) { +for_each_shadow_entry_lockless (vcpu, addr, iterator, spte)clear_sp_write_flooding_count (iterator.sptep); -if (!is_shadow_present_pte (spte)) -break ; - }walk_shadow_page_lockless_end (vcpu); }
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -982,7 +982,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)FNAME (update_pte)(vcpu, sp, sptep, &gpte); } -if (!is_shadow_present_pte (*sptep) || !sp->unsync_children) +if (!sp->unsync_children)break ; }write_unlock (&vcpu->kvm->mmu_lock);
Signature · 1 line
2.19.1.6.gb485710b
Re: [PATCH V3 2/2] KVM: X86: Move PTE present check from loop body to __shadow_walk_next()
Quoted 79 lines · Lai Jiangshan
But checking pte present in
__shadow_walk_next()is a more prudent way of programing and loop bodies will not need to always check it. It allows us removing unneededis_shadow_present_pte()in the loop bodies.Terminating on !is_shadow_present_pte() is 100% the correct behavior, as walking past a !PRESENT SPTE would lead to attempting to read a the next level SPTE from a garbage
iter->shadow_addr. Even some paths thatdo_not_currently have a !is_shadow_present_pte() in the loop body is Ok since they will install present non-leaf SPTEs and the additional present check is just an NOP.The checking result in
__shadow_walk_next()will be propagated toshadow_walk_okay()for being used in any for(;;) loop.Reviewed-by Sean Christopherson<[email protected]>Signed-off-by Lai Jiangshan<[email protected]>---
Changed from V2:
Fix typo in the changelog reported by Sean Add Reviewed-by from Sean
Changed from V1:
Merge the two patches
Update changelog
Remove !is_shadow_present_pte() in FNAME(invlpg)arch/x86/kvm/mmu/mmu.c | 13 ++----------- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 2 files changed, 3 insertions(+), 12 deletions(-)arch/x86/kvm/mmu/mmu.c +2
-11@@ -2223,7 +2223,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)static void __shadow_walk_next (struct kvm_shadow_walk_iterator *iterator,u64 spte) { -if (is_last_spte (spte, iterator->level)) { +if (!is_shadow_present_pte (spte) ||is_last_spte (spte, iterator->level)) { iterator->level =0 ;return ; }@@ -3159,9 +3159,6 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte)for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { sptep = iterator.sptep; *spte = old_spte; - -if (!is_shadow_present_pte (old_spte)) -break ; }return sptep;@@ -3721,9 +3718,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level spte =mmu_spte_get_lockless (iterator.sptep); sptes[leaf] = spte; - -if (!is_shadow_present_pte (spte)) -break ; }return leaf;@@ -3838,11 +3832,8 @@ static void shadow_page_table_clear_flood(struct kvm_vcpu *vcpu, gva_t addr)u64 spte;walk_shadow_page_lockless_begin (vcpu); -for_each_shadow_entry_lockless (vcpu, addr, iterator, spte) { +for_each_shadow_entry_lockless (vcpu, addr, iterator, spte)clear_sp_write_flooding_count (iterator.sptep); -if (!is_shadow_present_pte (spte)) -break ; - }walk_shadow_page_lockless_end (vcpu); }arch/x86/kvm/mmu/paging_tmpl.h +0
-0
arch/x86/kvm/mmu/paging_tmpl.h static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa) · line 985
Quoted 5 lines
arch/x86/kvm/mmu/paging_tmpl.h +1
-1@@ -982,7 +982,7 @@ static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva, hpa_t root_hpa)FNAME (update_pte)(vcpu, sp, sptep, &gpte); } -if (!is_shadow_present_pte (*sptep) || !sp->unsync_children) +if (!sp->unsync_children)
Queued both, thanks.
Paolo
Paolo Bonzini
maintainerRe: [PATCH V3 1/2] KVM: x86/mmu: Verify shadow walk doesn't terminate early in page faults
In reply to Lai JiangshanQuoted 3 lines · Lai Jiangshan
Opportunistically use fault->goal-level instead of it.level in FNAME(fetch) to further clarify that KVM always installs the leaf SPTE at the target level.
This argument will go away when mmu_set_spte starts fishing out the level from the kvm_mmu_page role, but it makes a good point for now.
Paolo
Lai Jiangshan
Acked-byQuoted 4 lines · Sean Christopherson
And for clarity and safety, I think it would be worth adding the patch below as a prep patch to document and enforce that walking the non-leaf SPTEs when faulting in a page should never terminate early.
I'v sent V2 patch which was updated as you suggested.
The patch you provided below doesn't need to be updated. It and my V2 patch do not depend on each other, so I didn't resent your patch with mine together.
For your patch
And it can be queued earlier.
Quoted 63 lines
From
1c202a7e82b1931e4eb37b23aa9d7108340a6cd2Mon Sep 17 00:00:00 2001
From: Sean Christopherson <[email protected]>
Date: Thu, 12 Aug 2021 08:38:35 -0700
Subject: [PATCH] KVM: x86/mmu: Verify shadow walk doesn't terminate early in
page faultsWARN and bail if the shadow walk for faulting in a SPTE terminates early, i.e. doesn't reach the expected level because the walk encountered a terminal SPTE. The shadow walks for page faults are subtle in that they install non-leaf SPTEs (zapping leaf SPTEs if necessary!) in the loop body, and consume the newly created non-leaf SPTE in the loop control, e.g.
__shadow_walk_next(). In other words, the walks guarantee that the walk will stop if and only if the target level is reached by installing non-leaf SPTEs to guarantee the walk remains valid.Opportunistically use fault->goal-level instead of it.level in FNAME(fetch) to further clarify that KVM always installs the leaf SPTE at the target level.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 3 +++ arch/x86/kvm/mmu/paging_tmpl.h | 7 +++++-- 2 files changed, 8 insertions(+), 2 deletions(-)arch/x86/kvm/mmu/mmu.c +3
-0@@ -2992,6 +2992,9 @@ static int __direct_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)account_huge_nx_page (vcpu->kvm, sp); } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep,ACC_ALL , fault->write, fault->goal_level, base_gfn, fault->pfn, fault->prefault, fault->map_writable);arch/x86/kvm/mmu/paging_tmpl.h +5
-3@@ -749,9 +749,12 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, } } +if (WARN_ON_ONCE (it.level != fault->goal_level)) +return -EFAULT ; + ret =mmu_set_spte (vcpu, it.sptep, gw->pte_access, fault->write, - it.level, base_gfn, fault->pfn, fault->prefault, - fault->map_writable); + fault->goal_level, base_gfn, fault->pfn, + fault->prefault, fault->map_writable);if (ret ==RET_PF_SPURIOUS )return ret; --2.33.0.rc1.237.g0d66db33f3-goog