KVM: x86/mmu: Fast page fault support for the TDP MMU
v3 cover KVM: x86/mmu: Fast page fault support for the TDP MMU
This patch series adds support for the TDP MMU in the fast_page_fault path, which enables certain write-protection and access tracking faults to be handled without taking the KVM MMU lock. This series brings the performance of these faults up to par with the legacy MMU.
Since there is not currently any KVM test coverage for access tracking faults, this series introduces a new KVM selftest, access_tracking_perf_test. This test relies on page_idle to enable access tracking from userspace (since it is the only available usersapce API to do so).
Matthew Wilcox, Yu Zhao, David Hildenbrand, and Andrew Morton: You are cc'd since you have discussed dropping page_idle from Linux [1].
Design
------
This series enables the existing fast_page_fault handler to operate independent of whether the TDP MMU is enabled or not by abstracting out the details behind a new lockless page walk API.
An alternative design considered was to add a separate fast_page_fault handler to the TDP MMU. The code that inspects the spte and genereates the new spte can be shared with the legacy MMU. However with this design the retry loop has to be duplicated, there are many calls back and forth between mmu.c and tdp_mmu.c, and passing around the RET_PF_* values gets complicated.
Testing
-------
This series was tested on an Intel Cascade Lake machine. The kvm_intel parameters eptad and pml were disabled to force access and dirty tracking to go through fast_page_fault. All tests were run with the TDP MMU enabled and then again disabled.
Tests ran:
- All KVM selftests with default arguments
- All x86_64 kvm-unit-tests.
- ./access_tracking_perf_test -v 4
- ./access_tracking_perf_test -v 4 -o
- ./access_tracking_perf_test -v 4 -s anonymous_thp
- ./access_tracking_perf_test -v 4 -s anonymous_thp -o
- ./access_tracking_perf_test -v 64
- ./dirty_log_perf_test -v 4
- ./dirty_log_perf_test -v 4 -o
- ./dirty_log_perf_test -v 4 -s anonymous_thp
- ./dirty_log_perf_test -v 4 -s anonymous_thp -o
- ./dirty_log_perf_test -v 64
For certain tests I also collected the fast_page_fault tracepoint to manually make sure it was getting triggered properly:
perf record -e kvmmmu:fast_page_fault --filter "old_spte != 0" -- <test>
Performance Results
-------------------
To measure performance I ran dirty_log_perf_test and access_tracking_perf_test with 64 vCPUs. For dirty_log_perf_test performance is measured by "Iteration 2 dirty memory time", the time it takes for all vCPUs to write to their memory after it has been write-protected. For access_tracking_perf_test performance is measured by "Writing to idle memory", the time it takes for all vCPUs to write to their memory after it has been access-protected.
Metric | tdp_mmu=Y before | tdp_mmu=Y after --------------------------------- | ------------------ | ----------------- Iteration 2 dirty memory time | 3.545234984s | 0.313867232s Writing to idle memory | 3.249645416s | 0.296113187s
The performance improvement comes from less time spent acquiring the mmu lock in read mode and less time looking up the memslot for the faulting gpa.
The TDP MMU is now on par with the legacy MMU:
Metric | tdp_mmu=N | tdp_mmu=Y --------------------------------- | ------------------ | ----------------- Iteration 2 dirty memory time | 0.303452990s | 0.313867232s Writing to idle memory | 0.291742127s | 0.296113187s
v3:
* PATCH 1/6: Add Sean's Reviewed-by.
* PATCH 2/6: Add TRACE_DEFINE_ENUM for all RET_PF_* values. [Ben]
* PATCH 2/6: Add comment for future RET_PF values. [me]
* PATCH 3/6: Pull walk_shadow_page_lockless_{begin,end} out of get_walk. [Ben]
* PATCH 3/6: Make kvm_tdp_mmu_walk_lockless_{begin,end} static inline. [Sean]
* PATCH 4/6: Make get_last_sptep_lockless static. [kernel test robot]
* PATCH 4/6: Fix comment above kvm_tdp_mmu_get_last_sptep_lockless. [me]
* PATCH 4/6: Rename and comment functions only meant for fast_page_fault handling. [Ben]
* PATCH 4/6: Improve comment in tdp_mmu_set_spte_atomic_no_dirty_log. [Sean]
* PATCH 4/6: Remove unnecessary sptep null check. [Sean]
v2: https://lore.kernel.org/kvm/[email protected]/
* Split is_tdp_mmu_root cleanup into a separate series. [Sean]
https://lore.kernel.org/kvm/[email protected]/
* Split walk_shadow_page_lockless into 2 APIs. [Sean]
* Perform rcu_dereference on TDP MMU sptep.
* Add comment to tdp_mmu_set_spte_atomic explaining new interaction
* with fast_pf_fix_direct_spte. [Ben]
* Document pagemap shifts in access_tracking_perf_test. [Ben]
* Skip test if lacking pagemap permissions (present pfn is 0). [Ben]
* Add Ben's Reviewed-by tags.
v1: https://lore.kernel.org/kvm/[email protected]/
[1] https://lore.kernel.org/linux-mm/[email protected]/
David Matlack (6):
KVM: x86/mmu: Rename cr2_or_gpa to gpa in fast_page_fault
KVM: x86/mmu: Fix use of enums in trace_fast_page_fault
KVM: x86/mmu: Make walk_shadow_page_lockless_{begin,end} interoperate
with the TDP MMU
KVM: x86/mmu: fast_page_fault support for the TDP MMU
KVM: selftests: Fix missing break in dirty_log_perf_test arg parsing
KVM: selftests: Introduce access_tracking_perf_test
arch/x86/kvm/mmu/mmu.c | 74 ++- arch/x86/kvm/mmu/mmu_internal.h | 3 + arch/x86/kvm/mmu/mmutrace.h | 6 + arch/x86/kvm/mmu/tdp_mmu.c | 47 +- arch/x86/kvm/mmu/tdp_mmu.h | 12 + tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 429 ++++++++++++++++++ .../selftests/kvm/dirty_log_perf_test.c | 1 + 9 files changed, 550 insertions(+), 24 deletions(-) create mode 100644 tools/testing/selftests/kvm/access_tracking_perf_test.c
Signature · 1 line
2.32.0.93.g670b81a890-goog
PATCH v3 1/6 KVM: x86/mmu: Rename cr2_or_gpa to gpa in fast_page_fault
fast_page_fault is only called from direct_page_fault where we know the address is a gpa.
--- arch/x86/kvm/mmu/mmu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -5
@@ -3098,8 +3098,7 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte)/* * Returns one of RET_PF_INVALID, RET_PF_FIXED or RET_PF_SPURIOUS. */ -static int fast_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa, -u32 error_code) +static int fast_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code) {struct kvm_shadow_walk_iterator iterator;struct kvm_mmu_page *sp;
@@ -3115,7 +3114,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, do { u64 new_spte; - for_each_shadow_entry_lockless(vcpu, cr2_or_gpa, iterator, spte) + for_each_shadow_entry_lockless(vcpu, gpa, iterator, spte) if (!is_shadow_present_pte(spte)) break;
@@ -3194,8 +3193,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, }while (true); -trace_fast_page_fault (vcpu, cr2_or_gpa, error_code, iterator.sptep, - spte, ret); +trace_fast_page_fault (vcpu, gpa, error_code, iterator.sptep, spte, ret);walk_shadow_page_lockless_end (vcpu);return ret;
Signature · 1 line
2.32.0.93.g670b81a890-goog
PATCH v3 2/6 KVM: x86/mmu: Fix use of enums in trace_fast_page_fault
In reply to David MatlackEnum values have to be exported to userspace since the formatting is not done in the kernel. Without doing this perf maps RET_PF_FIXED and RET_PF_SPURIOUS to 0, which results in incorrect output:
$ perf record -a -e kvmmmu:fast_page_fault --filter "ret==3" -- ./access_tracking_perf_test $ perf script | head -1 [...] new 610006048d25877 spurious 0 fixed 0 <------ should be 1
Fix this by exporting the enum values to userspace with TRACE_DEFINE_ENUM.
--- arch/x86/kvm/mmu/mmu_internal.h | 3 +++ arch/x86/kvm/mmu/mmutrace.h | 6 ++++++ 2 files changed, 9 insertions(+)
arch/x86/kvm/mmu/mmu_internal.h +3 -0
@@ -140,6 +140,9 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,* RET_PF_INVALID: the spte is invalid, let the real page fault path update it. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. +* +* Any names added to this enum should be exported to userspace for use in +* tracepoints via TRACE_DEFINE_ENUM() in mmutrace.h */ enum {RET_PF_RETRY =0 ,
arch/x86/kvm/mmu/mmutrace.h +6 -0
@@ -54,6 +54,12 @@ { PFERR_RSVD_MASK, "RSVD" }, \ { PFERR_FETCH_MASK, "F" } +TRACE_DEFINE_ENUM (RET_PF_RETRY ); +TRACE_DEFINE_ENUM (RET_PF_EMULATE ); +TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_FIXED ); +TRACE_DEFINE_ENUM (RET_PF_SPURIOUS ); +/* * A pagetable walk has started */
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH v3 2/6] KVM: x86/mmu: Fix use of enums in trace_fast_page_fault
Quoted 5 lines · David Matlack
Enum values have to be exported to userspace since the formatting is not done in the kernel. Without doing this perf maps
RET_PF_FIXEDandRET_PF_SPURIOUSto 0, which results in incorrect output:
Quoted 7 lines
$ perf record -a -e kvmmmu:fast_page_fault --filter "ret==3" -- ./access_tracking_perf_test $ perf script | head -1 [...] new 610006048d25877 spurious 0 fixed 0 <------ should be 1Fix this by exporting the enum values to userspace with
TRACE_DEFINE_ENUM.Fixes c4371c2a682e ("KVM: x86/mmu: Return unique RET_PF_* values if the fault was fixed")
Quoted 40 lines
Signed-off-by David Matlack <[email protected]>--- arch/x86/kvm/mmu/mmu_internal.h | 3 +++ arch/x86/kvm/mmu/mmutrace.h | 6 ++++++ 2 files changed, 9 insertions(+)arch/x86/kvm/mmu/mmu_internal.h +3
-0@@ -140,6 +140,9 @@ void kvm_flush_remote_tlbs_with_address(struct kvm *kvm,* RET_PF_INVALID: the spte is invalid, let the real page fault path update it. * RET_PF_FIXED: The faulting entry has been fixed. * RET_PF_SPURIOUS: The faulting entry was already fixed, e.g. by another vCPU. +* +* Any names added to this enum should be exported to userspace for use in +* tracepoints via TRACE_DEFINE_ENUM() in mmutrace.h */ enum {RET_PF_RETRY =0 ,arch/x86/kvm/mmu/mmutrace.h +6
-1@@ -54,6 +54,12 @@ { PFERR_RSVD_MASK, "RSVD" }, \ { PFERR_FETCH_MASK, "F" } +TRACE_DEFINE_ENUM (RET_PF_RETRY ); +TRACE_DEFINE_ENUM (RET_PF_EMULATE ); +TRACE_DEFINE_ENUM (RET_PF_INVALID ); +TRACE_DEFINE_ENUM (RET_PF_FIXED ); +TRACE_DEFINE_ENUM (RET_PF_SPURIOUS ); +/* * A pagetable walk has started */ --2.32.0.93.g670b81a890-goog
PATCH v3 3/6 KVM: x86/mmu: Make walk_shadow_page_lockless_{begin,end} interoperate with the TDP MMU
In reply to David MatlackAcquire the RCU read lock in walk_shadow_page_lockless_begin and release it in walk_shadow_page_lockless_end when the TDP MMU is enabled. This should not introduce any functional changes but is used in the following commit to make fast_page_fault interoperate with the TDP MMU.
--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.c | 6 ++---- arch/x86/kvm/mmu/tdp_mmu.h | 10 ++++++++++ 3 files changed, 28 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu/mmu.c +16 -4
@@ -686,6 +686,11 @@ static bool mmu_spte_age(u64 *sptep)static void walk_shadow_page_lockless_begin (struct kvm_vcpu *vcpu) { +if (is_tdp_mmu (vcpu->arch.mmu)) { +kvm_tdp_mmu_walk_lockless_begin (); +return ; + } +/* * Prevent page table teardown by making any free-er wait during * kvm_flush_remote_tlbs() IPI to all active vcpus.
@@ -701,6 +706,11 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)static void walk_shadow_page_lockless_end (struct kvm_vcpu *vcpu) { +if (is_tdp_mmu (vcpu->arch.mmu)) { +kvm_tdp_mmu_walk_lockless_end (); +return ; + } +/* * Make sure the write to vcpu->mode is not reordered in front of * reads to sptes. If it does, kvm_mmu_commit_zap_page() can see us
@@ -3612,6 +3622,8 @@ static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct)/* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present. +* +* Must be called between walk_shadow_page_lockless_{begin,end}. */ static int get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level) {
@@ -3619,8 +3631,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_levelint leaf =-1 ;u64 spte; -walk_shadow_page_lockless_begin (vcpu); - for (shadow_walk_init(&iterator, vcpu, addr), *root_level = iterator.level; shadow_walk_okay(&iterator);
@@ -3634,8 +3644,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_levelbreak ; } -walk_shadow_page_lockless_end (vcpu); -return leaf; }
@@ -3647,11 +3655,15 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep)int root, leaf, level;bool reserved = false; +walk_shadow_page_lockless_begin (vcpu); +if (is_tdp_mmu (vcpu->arch.mmu)) leaf =kvm_tdp_mmu_get_walk (vcpu, addr, sptes, &root);else leaf =get_walk (vcpu, addr, sptes, &root); +walk_shadow_page_lockless_end (vcpu); +if (unlikely (leaf <0 )) { *sptep =0ull ;return reserved;
arch/x86/kvm/mmu/tdp_mmu.c +2 -4
@@ -1516,6 +1516,8 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,/* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present. +* +* Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. */ int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level)
@@ -1527,14 +1529,10 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, *root_level = vcpu->arch.mmu->shadow_root_level; -rcu_read_lock (); -tdp_mmu_for_each_pte (iter, mmu, gfn, gfn +1 ) { leaf = iter.level; sptes[leaf] = iter.old_spte; } -rcu_read_unlock (); -return leaf; }
arch/x86/kvm/mmu/tdp_mmu.h +10 -0
@@ -77,6 +77,16 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, int min_level); +static inline void kvm_tdp_mmu_walk_lockless_begin (void ) +{ +rcu_read_lock (); +} + +static inline void kvm_tdp_mmu_walk_lockless_end (void ) +{ +rcu_read_unlock (); +} +int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level);
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH v3 3/6] KVM: x86/mmu: Make walk_shadow_page_lockless_{begin,end} interoperate with the TDP MMU
Quoted 6 lines · David Matlack
Acquire the RCU read lock in
walk_shadow_page_lockless_beginand release it inwalk_shadow_page_lockless_endwhen the TDP MMU is enabled. This should not introduce any functional changes but is used in the following commit to makefast_page_faultinteroperate with the TDP MMU.
Quoted 130 lines
Signed-off-by David Matlack <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 20 ++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.c | 6 ++---- arch/x86/kvm/mmu/tdp_mmu.h | 10 ++++++++++ 3 files changed, 28 insertions(+), 8 deletions(-)arch/x86/kvm/mmu/mmu.c +16
-4@@ -686,6 +686,11 @@ static bool mmu_spte_age(u64 *sptep)static void walk_shadow_page_lockless_begin (struct kvm_vcpu *vcpu) { +if (is_tdp_mmu (vcpu->arch.mmu)) { +kvm_tdp_mmu_walk_lockless_begin (); +return ; + } +/* * Prevent page table teardown by making any free-er wait during * kvm_flush_remote_tlbs() IPI to all active vcpus. @@ -701,6 +706,11 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu)static void walk_shadow_page_lockless_end (struct kvm_vcpu *vcpu) { +if (is_tdp_mmu (vcpu->arch.mmu)) { +kvm_tdp_mmu_walk_lockless_end (); +return ; + } +/* * Make sure the write to vcpu->mode is not reordered in front of * reads to sptes. If it does, kvm_mmu_commit_zap_page() can see us @@ -3612,6 +3622,8 @@ static bool mmio_info_in_cache(struct kvm_vcpu *vcpu, u64 addr, bool direct)/* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present. +* +* Must be called between walk_shadow_page_lockless_{begin,end}. */ static int get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level) {@@ -3619,8 +3631,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_levelint leaf =-1 ;u64 spte; -walk_shadow_page_lockless_begin (vcpu); - for (shadow_walk_init(&iterator, vcpu, addr), *root_level = iterator.level; shadow_walk_okay(&iterator);@@ -3634,8 +3644,6 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_levelbreak ; } -walk_shadow_page_lockless_end (vcpu); -return leaf; }@@ -3647,11 +3655,15 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep)int root, leaf, level;bool reserved = false; +walk_shadow_page_lockless_begin (vcpu); +if (is_tdp_mmu (vcpu->arch.mmu)) leaf =kvm_tdp_mmu_get_walk (vcpu, addr, sptes, &root);else leaf =get_walk (vcpu, addr, sptes, &root); +walk_shadow_page_lockless_end (vcpu); +if (unlikely (leaf <0 )) { *sptep =0ull ;return reserved;arch/x86/kvm/mmu/tdp_mmu.c +2
-4@@ -1516,6 +1516,8 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,/* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present. +* +* Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. */ int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level)@@ -1527,14 +1529,10 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, *root_level = vcpu->arch.mmu->shadow_root_level; -rcu_read_lock (); -tdp_mmu_for_each_pte (iter, mmu, gfn, gfn +1 ) { leaf = iter.level; sptes[leaf] = iter.old_spte; } -rcu_read_unlock (); -return leaf; }arch/x86/kvm/mmu/tdp_mmu.h +10
-1@@ -77,6 +77,16 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, int min_level); +static inline void kvm_tdp_mmu_walk_lockless_begin (void ) +{ +rcu_read_lock (); +} + +static inline void kvm_tdp_mmu_walk_lockless_end (void ) +{ +rcu_read_unlock (); +} +int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level); --2.32.0.93.g670b81a890-goog
PATCH v3 4/6 KVM: x86/mmu: fast_page_fault support for the TDP MMU
In reply to David MatlackMake fast_page_fault interoperate with the TDP MMU by leveraging walk_shadow_page_lockless_{begin,end} to acquire the RCU read lock and introducing a new helper function kvm_tdp_mmu_fast_pf_get_last_sptep to grab the lowest level sptep.
--- arch/x86/kvm/mmu/mmu.c | 50 ++++++++++++++++++++++++++++---------- arch/x86/kvm/mmu/tdp_mmu.c | 41 +++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 2 ++ 3 files changed, 80 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +37 -13
@@ -3105,15 +3105,41 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte)return spte &PT_PRESENT_MASK ; } +/* +* Returns the last level spte pointer of the shadow page walk for the given +* gpa, and sets *spte to the spte value. This spte may be non-preset. If no +* walk could be performed, returns NULL and *spte does not contain valid data. +* +* Contract: +* - Must be called between walk_shadow_page_lockless_{begin,end}. +* - The returned sptep must not be used after walk_shadow_page_lockless_end. +*/ +static u64 *fast_pf_get_last_sptep (struct kvm_vcpu *vcpu,gpa_t gpa,u64 *spte) +{ +struct kvm_shadow_walk_iterator iterator; +u64 old_spte; +u64 *sptep =NULL ; + +for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { + sptep = iterator.sptep; + *spte = old_spte; + +if (!is_shadow_present_pte (old_spte)) +break ; + } + +return sptep; +} +/* * Returns one of RET_PF_INVALID, RET_PF_FIXED or RET_PF_SPURIOUS. */ static int fast_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code) { -struct kvm_shadow_walk_iterator iterator;struct kvm_mmu_page *sp;int ret =RET_PF_INVALID ;u64 spte =0ull ; +u64 *sptep =NULL ;uint retry_count =0 ;if (!page_fault_can_be_fast (error_code))
@@ -3124,14 +3150,15 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) do { u64 new_spte; - for_each_shadow_entry_lockless(vcpu, gpa, iterator, spte) - if (!is_shadow_present_pte(spte)) - break; + if (is_tdp_mmu(vcpu->arch.mmu)) + sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, gpa, &spte); + else + sptep = fast_pf_get_last_sptep(vcpu, gpa, &spte); if (!is_shadow_present_pte(spte)) break; - sp = sptep_to_sp(iterator.sptep); + sp = sptep_to_sp(sptep); if (!is_last_spte(spte, sp->role.level)) break;
@@ -3189,8 +3216,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code)* since the gfn is not stable for indirect shadow page. See * Documentation/virt/kvm/locking.rst to get more detail. */ -if (fast_pf_fix_direct_spte (vcpu, sp, iterator.sptep, spte, - new_spte)) { +if (fast_pf_fix_direct_spte (vcpu, sp, sptep, spte, new_spte)) { ret =RET_PF_FIXED ;break ; }
@@ -3203,7 +3229,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) }while (true); -trace_fast_page_fault (vcpu, gpa, error_code, iterator.sptep, spte, ret); +trace_fast_page_fault (vcpu, gpa, error_code, sptep, spte, ret);walk_shadow_page_lockless_end (vcpu);return ret;
@@ -3838,11 +3864,9 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (page_fault_handle_page_track (vcpu, error_code, gfn))return RET_PF_EMULATE ; -if (!is_tdp_mmu_fault) { - r =fast_page_fault (vcpu, gpa, error_code); -if (r !=RET_PF_INVALID ) -return r; - } + r =fast_page_fault (vcpu, gpa, error_code); +if (r !=RET_PF_INVALID ) +return r; r =mmu_topup_memory_caches (vcpu, false);if (r)
arch/x86/kvm/mmu/tdp_mmu.c +41 -0
50 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +2 -0
@@ -89,6 +89,8 @@ static inline void kvm_tdp_mmu_walk_lockless_end(void)int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level); +u64 *kvm_tdp_mmu_fast_pf_get_last_sptep (struct kvm_vcpu *vcpu,u64 addr, +u64 *spte);#ifdef CONFIG_X86_64 bool kvm_mmu_init_tdp_mmu (struct kvm *kvm);
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH v3 4/6] KVM: x86/mmu: fast_page_fault support for the TDP MMU
Quoted 8 lines · David Matlack
Make
fast_page_faultinteroperate with the TDP MMU by leveraging walk_shadow_page_lockless_{begin,end} to acquire the RCU read lock and introducing a new helper functionkvm_tdp_mmu_fast_pf_get_last_sptepto grab the lowest level sptep.Suggested-by Ben Gardon <[email protected]>Signed-off-by David Matlack <[email protected]>
Quoted 127 lines
--- arch/x86/kvm/mmu/mmu.c | 50 ++++++++++++++++++++++++++++---------- arch/x86/kvm/mmu/tdp_mmu.c | 41 +++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 2 ++ 3 files changed, 80 insertions(+), 13 deletions(-)arch/x86/kvm/mmu/mmu.c +37
-13@@ -3105,15 +3105,41 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte)return spte &PT_PRESENT_MASK ; } +/* +* Returns the last level spte pointer of the shadow page walk for the given +* gpa, and sets *spte to the spte value. This spte may be non-preset. If no +* walk could be performed, returns NULL and *spte does not contain valid data. +* +* Contract: +* - Must be called between walk_shadow_page_lockless_{begin,end}. +* - The returned sptep must not be used after walk_shadow_page_lockless_end. +*/ +static u64 *fast_pf_get_last_sptep (struct kvm_vcpu *vcpu,gpa_t gpa,u64 *spte) +{ +struct kvm_shadow_walk_iterator iterator; +u64 old_spte; +u64 *sptep =NULL ; + +for_each_shadow_entry_lockless (vcpu, gpa, iterator, old_spte) { + sptep = iterator.sptep; + *spte = old_spte; + +if (!is_shadow_present_pte (old_spte)) +break ; + } + +return sptep; +} +/* * Returns one of RET_PF_INVALID, RET_PF_FIXED or RET_PF_SPURIOUS. */ static int fast_page_fault (struct kvm_vcpu *vcpu,gpa_t gpa,u32 error_code) { -struct kvm_shadow_walk_iterator iterator;struct kvm_mmu_page *sp;int ret =RET_PF_INVALID ;u64 spte =0ull ; +u64 *sptep =NULL ;uint retry_count =0 ;if (!page_fault_can_be_fast (error_code))@@ -3124,14 +3150,15 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) do { u64 new_spte; - for_each_shadow_entry_lockless(vcpu, gpa, iterator, spte) - if (!is_shadow_present_pte(spte)) - break; + if (is_tdp_mmu(vcpu->arch.mmu)) + sptep = kvm_tdp_mmu_fast_pf_get_last_sptep(vcpu, gpa, &spte); + else + sptep = fast_pf_get_last_sptep(vcpu, gpa, &spte); if (!is_shadow_present_pte(spte)) break; - sp = sptep_to_sp(iterator.sptep); + sp = sptep_to_sp(sptep); if (!is_last_spte(spte, sp->role.level)) break;@@ -3189,8 +3216,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code)* since the gfn is not stable for indirect shadow page. See * Documentation/virt/kvm/locking.rst to get more detail. */ -if (fast_pf_fix_direct_spte (vcpu, sp, iterator.sptep, spte, - new_spte)) { +if (fast_pf_fix_direct_spte (vcpu, sp, sptep, spte, new_spte)) { ret =RET_PF_FIXED ;break ; }@@ -3203,7 +3229,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) }while (true); -trace_fast_page_fault (vcpu, gpa, error_code, iterator.sptep, spte, ret); +trace_fast_page_fault (vcpu, gpa, error_code, sptep, spte, ret);walk_shadow_page_lockless_end (vcpu);return ret;@@ -3838,11 +3864,9 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (page_fault_handle_page_track (vcpu, error_code, gfn))return RET_PF_EMULATE ; -if (!is_tdp_mmu_fault) { - r =fast_page_fault (vcpu, gpa, error_code); -if (r !=RET_PF_INVALID ) -return r; - } + r =fast_page_fault (vcpu, gpa, error_code); +if (r !=RET_PF_INVALID ) +return r; r =mmu_topup_memory_caches (vcpu, false);if (r)arch/x86/kvm/mmu/tdp_mmu.c +5
-0@@ -527,6 +527,10 @@ static inline bool tdp_mmu_set_spte_atomic_no_dirty_log(struct kvm *kvm,if (is_removed_spte (iter->old_spte))return false; +/* +* Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and +* does not hold the mmu_lock. +*/ if (cmpxchg64 (rcu_dereference (iter->sptep), iter->old_spte, new_spte) != iter->old_spte)return false;@@ -1536,3 +1540,40 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,return leaf; } +
arch/x86/kvm/mmu/tdp_mmu.c int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, · line 1553
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10
-0+/* +* Returns the last level spte pointer of the shadow page walk for the given +* gpa, and sets *spte to the spte value. This spte may be non-preset. If no +* walk could be performed, returns NULL and *spte does not contain valid data. +* +* Contract: +* - Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. +* - The returned sptep must not be used after kvm_tdp_mmu_walk_lockless_end. +* +* WARNING: This function is only intended to be called during fast_page_fault.
Heh, this warning seems a bit overkill, but no harm there.
Quoted 42 lines
arch/x86/kvm/mmu/tdp_mmu.c +26
-0+*/ +u64 *kvm_tdp_mmu_fast_pf_get_last_sptep (struct kvm_vcpu *vcpu,u64 addr, +u64 *spte) +{ +struct tdp_iter iter; +struct kvm_mmu *mmu = vcpu->arch.mmu; +gfn_t gfn = addr >>PAGE_SHIFT ; +tdp_ptep_t sptep =NULL ; + +tdp_mmu_for_each_pte (iter, mmu, gfn, gfn +1 ) { + *spte = iter.old_spte; + sptep = iter.sptep; + } + +/* +* Perform the rcu_dereference to get the raw spte pointer value since +* we are passing it up to fast_page_fault, which is shared with the +* legacy MMU and thus does not retain the TDP MMU-specific __rcu +* annotation. +* +* This is safe since fast_page_fault obeys the contracts of this +* function as well as all TDP MMU contracts around modifying SPTEs +* outside of mmu_lock. +*/ +return rcu_dereference (sptep); +}arch/x86/kvm/mmu/tdp_mmu.h +2
-1@@ -89,6 +89,8 @@ static inline void kvm_tdp_mmu_walk_lockless_end(void)int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes,int *root_level); +u64 *kvm_tdp_mmu_fast_pf_get_last_sptep (struct kvm_vcpu *vcpu,u64 addr, +u64 *spte);#ifdef CONFIG_X86_64 bool kvm_mmu_init_tdp_mmu (struct kvm *kvm); --2.32.0.93.g670b81a890-goog
PATCH v3 5/6 KVM: selftests: Fix missing break in dirty_log_perf_test arg parsing
In reply to David MatlackThere is a missing break statement which causes a fallthrough to the next statement where optarg will be null and a segmentation fault will be generated.
--- tools/testing/selftests/kvm/dirty_log_perf_test.c | 1 + 1 file changed, 1 insertion(+)
tools/testing/selftests/kvm/dirty_log_perf_test.c +1 -0
@@ -312,6 +312,7 @@ int main(int argc, char *argv[])break ;case 'o' : p.partition_vcpu_memory_access = false; +break ;case 's' : p.backing_src =parse_backing_src_type (optarg);break ;
Signature · 1 line
2.32.0.93.g670b81a890-goog
PATCH v3 6/6 KVM: selftests: Introduce access_tracking_perf_test
In reply to David MatlackThis test measures the performance effects of KVM's access tracking. Access tracking is driven by the MMU notifiers test_young, clear_young, and clear_flush_young. These notifiers do not have a direct userspace API, however the clear_young notifier can be triggered by marking a pages as idle in /sys/kernel/mm/page_idle/bitmap. This test leverages that mechanism to enable access tracking on guest memory.
To measure performance this test runs a VM with a configurable number of vCPUs that each touch every page in disjoint regions of memory. Performance is measured in the time it takes all vCPUs to finish touching their predefined region.
Example invocation:
$ ./access_tracking_perf_test -v 8 Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffdfffff000
Populating memory : 1.337752570s
Writing to populated memory : 0.010177640s
Reading from populated memory : 0.009548239s
Mark memory idle : 23.973131748s Writing to idle memory : 0.063584496s Mark memory idle : 24.924652964s Reading from idle memory : 0.062042814s
Breaking down the results:
* "Populating memory": The time it takes for all vCPUs to perform the first write to every page in their region.
* "Writing to populated memory" / "Reading from populated memory": The time it takes for all vCPUs to write and read to every page in their region after it has been populated. This serves as a control for the later results.
* "Mark memory idle": The time it takes for every vCPU to mark every page in their region as idle through page_idle.
* "Writing to idle memory" / "Reading from idle memory": The time it takes for all vCPUs to write and read to every page in their region after it has been marked idle.
This test should be portable across architectures but it is only enabled for x86_64 since that's all I have tested.
--- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 429 ++++++++++++++++++ 3 files changed, 431 insertions(+) create mode 100644 tools/testing/selftests/kvm/access_tracking_perf_test.c
tools/testing/selftests/kvm/.gitignore +1 -0
@@ -38,6 +38,7 @@ /x86_64/xen_vmcall_test /x86_64/xss_msr_test /x86_64/vmx_pmu_msrs_test +/access_tracking_perf_test /demand_paging_test /dirty_log_test /dirty_log_perf_test
tools/testing/selftests/kvm/Makefile +1 -0
@@ -71,6 +71,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/tsc_msrs_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_pmu_msrs_test TEST_GEN_PROGS_x86_64 += x86_64/xen_shinfo_test TEST_GEN_PROGS_x86_64 += x86_64/xen_vmcall_test +TEST_GEN_PROGS_x86_64 += access_tracking_perf_test TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test TEST_GEN_PROGS_x86_64 += dirty_log_perf_test
tools/testing/selftests/kvm/access_tracking_perf_test.c new file +429 -0
429 lines not on this page · raw message
Signature · 1 line
2.32.0.93.g670b81a890-goog
Quoted 4 lines · David Matlack
This patch series adds support for the TDP MMU in the
fast_page_faultpath, which enables certain write-protection and access tracking faults to be handled without taking the KVM MMU lock. This series brings the performance of these faults up to par with the legacy MMU.
Queued, thanks.
Paolo
Quoted 7 lines · Paolo Bonzini
Quoted 4 lines · David Matlack
This patch series adds support for the TDP MMU in the
fast_page_faultpath, which enables certain write-protection and access tracking faults to be handled without taking the KVM MMU lock. This series brings the performance of these faults up to par with the legacy MMU.Queued, thanks.
Thanks Paolo. I noticed on PATCH 3 you changed the if-return to if-else. What was your rational? (I want to make sure I incorporate it in future patches.)
Quoted 3 lines
Paolo