Improve gfn-to-memslot performance during page faults
cover Improve gfn-to-memslot performance during page faults
This series improves the performance of gfn-to-memslot lookups during page faults. Ben Gardon originally identified this performance gap and sufficiently addressed it in Google's kernel by reading the memslot once at the beginning of the page fault and passing around the pointer.
This series takes an alternative approach by introducing a per-vCPU cache of the least recently used memslot index. This avoids needing to binary search the existing memslots multiple times during a page fault. Unlike passing around the pointer, the LRU cache has an additional benefit in that it speeds up gfn-to-memslot lookups *across* faults and during spte prefetching where the gfn changes.
This difference can be seen clearly when looking at the performance of fast_page_fault when multiple slots are in play:
Metric | Baseline | Pass* | LRU** ----------------------------- | ------------ | -------- | ---------- Iteration 2 dirty memory time | 2.8s | 1.6s | 0.30s
* Pass: Lookup the memslot once per fault and pass it around. ** LRU: Cache the LRU slot per vCPU (i.e. this series).
(Collected via ./dirty_log_perf_test -v64 -x64)
I plan to also send a follow-up series with a version of Ben's patches to pass the pointer to the memslot through the page fault handling code rather than looking it up multiple times. Even when applied on top of the LRU series it has some performance improvements by avoiding a few extra memory accesses (mainly kvm->memslots[as_id] and slots->used_slots). But it will be a judgement call whether or not it's worth the code churn and complexity.
Here is a break down of this series:
Patches 1-2 introduce a per-vCPU cache of the least recently memslot index.
Patches 3-5 convert existing gfn-to-memslot lookups to use kvm_vcpu_gfn_to_memslot so that they can leverage the new LRU cache.
Patch 6 adds support for multiple slots to dirty_log_perf_test which is used to generate the performance data in this series.
David Matlack (6):
KVM: Cache the least recently used slot index per vCPU
KVM: Avoid VM-wide lru_slot lookup in kvm_vcpu_gfn_to_memslot
KVM: x86/mmu: Speed up dirty logging in
tdp_mmu_map_handle_target_level
KVM: x86/mmu: Leverage vcpu->lru_slot_index for rmap_add and
rmap_recycle
KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap
KVM: selftests: Support multiple slots in dirty_log_perf_test
arch/x86/kvm/mmu/mmu.c | 54 +++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 15 +++- include/linux/kvm_host.h | 73 +++++++++++++----- .../selftests/kvm/access_tracking_perf_test.c | 2 +- .../selftests/kvm/demand_paging_test.c | 2 +- .../selftests/kvm/dirty_log_perf_test.c | 76 ++++++++++++++++--- .../selftests/kvm/include/perf_test_util.h | 2 +- .../selftests/kvm/lib/perf_test_util.c | 20 +++-- .../kvm/memslot_modification_stress_test.c | 2 +- virt/kvm/kvm_main.c | 21 ++++- 10 files changed, 198 insertions(+), 69 deletions(-)
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
PATCH 1/6 KVM: Cache the least recently used slot index per vCPU
The memslot for a given gfn is looked up multiple times during page fault handling. Avoid binary searching for it multiple times by caching the least recently used slot. There is an existing VM-wide LRU slot but that does not work well for cases where vCPUs are accessing memory in different slots (see performance data below).
Another benefit of caching the least recently use slot (versus looking up the slot once and passing around a pointer) is speeding up memslot lookups *across* faults and during spte prefetching.
To measure the performance of this change I ran dirty_log_perf_test with 64 vCPUs and 64 memslots and measured "Populate memory time" and "Iteration 2 dirty memory time". Tests were ran with eptad=N to force dirty logging to use fast_page_fault so its performance could be measured.
Config | Metric | Before | After ---------- | ----------------------------- | ------ | ------ tdp_mmu=Y | Populate memory time | 6.76s | 5.47s tdp_mmu=Y | Iteration 2 dirty memory time | 2.83s | 0.31s tdp_mmu=N | Populate memory time | 20.4s | 18.7s tdp_mmu=N | Iteration 2 dirty memory time | 2.65s | 0.30s
The "Iteration 2 dirty memory time" results are especially compelling because they are equivalent to running the same test with a single memslot. In other words, fast_page_fault performance no longer scales with the number of memslots.
--- include/linux/kvm_host.h | 61 ++++++++++++++++++++++++++++++---------- virt/kvm/kvm_main.c | 21 +++++++++++++- 2 files changed, 66 insertions(+), 16 deletions(-)
include/linux/kvm_host.h +46 -15
@@ -354,6 +354,13 @@ struct kvm_vcpu {struct kvm_vcpu_stat stat;char stats_id[KVM_STATS_NAME_SIZE ];struct kvm_dirty_ring dirty_ring; + +/* +* The index of the least recently used memslot by this vCPU. It's ok +* if this becomes stale due to memslot changes since we always check +* it is a valid slot. +*/ +int lru_slot_index; };/* must be called with irqs disabled */
@@ -1189,27 +1196,38 @@ int kvm_request_irq_source_id(struct kvm *kvm);void kvm_free_irq_source_id (struct kvm *kvm,int irq_source_id);bool kvm_arch_irqfd_allowed (struct kvm *kvm,struct kvm_irqfd *args); +static inline struct kvm_memory_slot *get_slot (struct kvm_memslots *slots,int slot_index) +{ +if (slot_index <0 || slot_index >= slots->used_slots) +return NULL ; + +return &slots->memslots[slot_index]; +} + +static inline bool slot_contains_gfn (struct kvm_memslots *slots,int slot_index,gfn_t gfn) +{ +struct kvm_memory_slot *memslot =get_slot (slots, slot_index); + +if (!memslot) +return false; + +return gfn >= memslot->base_gfn && gfn < memslot->base_gfn + memslot->npages; +} +/* -* search_memslots() and __gfn_to_memslot() are here because they are -* used in non-modular code in arch/powerpc/kvm/book3s_hv_rm_mmu.c. -* gfn_to_memslot() itself isn't here as an inline because that would -* bloat other code too much. -* * IMPORTANT: Slots are sorted from highest GFN to lowest GFN! */ -static inline struct kvm_memory_slot * -search_memslots (struct kvm_memslots *slots,gfn_t gfn) +static inline int __search_memslots (struct kvm_memslots *slots,gfn_t gfn) {int start =0 , end = slots->used_slots;int slot =atomic_read (&slots->lru_slot);struct kvm_memory_slot *memslots = slots->memslots;if (unlikely (!slots->used_slots)) -return NULL ; +return -1 ; -if (gfn >= memslots[slot].base_gfn && - gfn < memslots[slot].base_gfn + memslots[slot].npages) -return &memslots[slot]; +if (slot_contains_gfn (slots, slot, gfn)) +return slot;while (start < end) { slot = start + (end - start) /2 ;
@@ -1220,13 +1238,26 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn) start = slot +1 ; } -if (start < slots->used_slots && gfn >= memslots[start].base_gfn && - gfn < memslots[start].base_gfn + memslots[start].npages) { +if (slot_contains_gfn (slots, start, gfn)) {atomic_set (&slots->lru_slot, start); -return &memslots[start]; +return start; } -return NULL ; +return -1 ; +} + +/* +* search_memslots() and __gfn_to_memslot() are here because they are +* used in non-modular code in arch/powerpc/kvm/book3s_hv_rm_mmu.c. +* gfn_to_memslot() itself isn't here as an inline because that would +* bloat other code too much. +*/ +static inline struct kvm_memory_slot * +search_memslots (struct kvm_memslots *slots,gfn_t gfn) +{ +int slot_index =__search_memslots (slots, gfn); + +return get_slot (slots, slot_index); } static inline struct kvm_memory_slot *
virt/kvm/kvm_main.c +20 -1
@@ -415,6 +415,7 @@ static void kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id) vcpu->preempted = false; vcpu->ready = false;preempt_notifier_init (&vcpu->preempt_notifier, &kvm_preempt_ops); + vcpu->lru_slot_index =0 ; }void kvm_vcpu_destroy (struct kvm_vcpu *vcpu)
@@ -2024,7 +2025,25 @@ EXPORT_SYMBOL_GPL(gfn_to_memslot);struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot (struct kvm_vcpu *vcpu,gfn_t gfn) { -return __gfn_to_memslot (kvm_vcpu_memslots (vcpu), gfn); +struct kvm_memslots *slots =kvm_vcpu_memslots (vcpu); +int slot_index = vcpu->lru_slot_index; +struct kvm_memory_slot *slot; + +if (!slot_contains_gfn (slots, slot_index, gfn)) + slot_index =__search_memslots (slots, gfn); + + slot =get_slot (slots, slot_index); + +/* +* Purposely avoid updating vcpu->lru_slot_index if the gfn is not +* backed by memslot as that will guarantee a cache miss on the next +* try. By leaving vcpu->lru_slot_index untouched we have a chance of +* a hit on the next lookup. +*/ +if (slot) + vcpu->lru_slot_index = slot_index; + +return slot; }EXPORT_SYMBOL_GPL (kvm_vcpu_gfn_to_memslot);
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
Paolo Bonzini
maintainerRe: [PATCH 1/6] KVM: Cache the least recently used slot index per vCPU
Quoted 23 lines · David Matlack
The memslot for a given gfn is looked up multiple times during page fault handling. Avoid binary searching for it multiple times by caching the least recently used slot. There is an existing VM-wide LRU slot but that does not work well for cases where vCPUs are accessing memory in different slots (see performance data below).
Another benefit of caching the least recently use slot (versus looking up the slot once and passing around a pointer) is speeding up memslot lookups *across* faults and during spte prefetching.
To measure the performance of this change I ran
dirty_log_perf_testwith 64 vCPUs and 64 memslots and measured "Populate memory time" and "Iteration 2 dirty memory time". Tests were ran with eptad=N to force dirty logging to usefast_page_faultso its performance could be measured.Config | Metric | Before | After ---------- | ----------------------------- | ------ | ------ tdp_mmu=Y | Populate memory time | 6.76s | 5.47s tdp_mmu=Y | Iteration 2 dirty memory time | 2.83s | 0.31s tdp_mmu=N | Populate memory time | 20.4s | 18.7s tdp_mmu=N | Iteration 2 dirty memory time | 2.65s | 0.30s
Quoted 6 lines
The "Iteration 2 dirty memory time" results are especially compelling because they are equivalent to running the same test with a single memslot. In other words,
fast_page_faultperformance no longer scales with the number of memslots.Signed-off-by David Matlack <[email protected]>
It's the *most* recently used slot index, of course. :) That's true of lru_slot as well.
include/linux/kvm_host.h int kvm_request_irq_source_id(struct kvm *kvm); · line 1206
Quoted 8 lines
include/linux/kvm_host.h +8
-0+static inline struct kvm_memory_slot *get_slot (struct kvm_memslots *slots,int slot_index) +{ +if (slot_index <0 || slot_index >= slots->used_slots) +return NULL ; + +return &slots->memslots[slot_index]; +} +
Since there are plenty of arrays inside struct kvm_memory_slot*, do we want to protect this against speculative out-of-bounds accesses with array_index_nospec?
include/linux/kvm_host.h search_memslots(struct kvm_memslots *slots, gfn_t gfn) · line 1261
Quoted 7 lines
include/linux/kvm_host.h +6
-0+static inline struct kvm_memory_slot * +search_memslots (struct kvm_memslots *slots,gfn_t gfn) +{ +int slot_index =__search_memslots (slots, gfn); + +return get_slot (slots, slot_index); }
Let's use this occasion to remove the duplication between __gfn_to_memslot and search_memslots; you can make search_memslots do the search and palce the LRU (ehm, MRU) code to __gfn_to_memslot only. So:
- the new patch 1 (something like "make search_memslots search without LRU caching") is basically this series's patch 2, plus a tree-wide replacement of search_memslots with __gfn_to_memslot.
- the new patch 2 is this series's patch 1, except kvm_vcpu_gfn_to_memslot uses search_memslots instead of __search_memslots. The comments in patch 2's commit message about the double misses move to this commit message.
virt/kvm/kvm_main.c EXPORT_SYMBOL_GPL(gfn_to_memslot); · line 2044
Quoted 2 lines
virt/kvm/kvm_main.c +2
-0+if (slot) + vcpu->lru_slot_index = slot_index;
Let's call it lru_slot for consistency with the field of struct kvm_memory_slots.
Paolo
Re: [PATCH 1/6] KVM: Cache the least recently used slot index per vCPU
Quoted 29 lines · Paolo Bonzini
Quoted 28 lines · David Matlack
The memslot for a given gfn is looked up multiple times during page fault handling. Avoid binary searching for it multiple times by caching the least recently used slot. There is an existing VM-wide LRU slot but that does not work well for cases where vCPUs are accessing memory in different slots (see performance data below).
Another benefit of caching the least recently use slot (versus looking up the slot once and passing around a pointer) is speeding up memslot lookups *across* faults and during spte prefetching.
To measure the performance of this change I ran
dirty_log_perf_testwith 64 vCPUs and 64 memslots and measured "Populate memory time" and "Iteration 2 dirty memory time". Tests were ran with eptad=N to force dirty logging to usefast_page_faultso its performance could be measured.Config | Metric | Before | After ---------- | ----------------------------- | ------ | ------ tdp_mmu=Y | Populate memory time | 6.76s | 5.47s tdp_mmu=Y | Iteration 2 dirty memory time | 2.83s | 0.31s tdp_mmu=N | Populate memory time | 20.4s | 18.7s tdp_mmu=N | Iteration 2 dirty memory time | 2.65s | 0.30sThe "Iteration 2 dirty memory time" results are especially compelling because they are equivalent to running the same test with a single memslot. In other words,
fast_page_faultperformance no longer scales with the number of memslots.
Quoted 4 lines
Quoted 1 line
Signed-off-by David Matlack <[email protected]>It's the *most* recently used slot index, of course. :) That's true of
lru_slotas well.
*facepalm*
I'll include a patch in v2 to fix the name of the existing lru_slot to something like mru_slot or last_used_slot.
Quoted 13 lines
include/linux/kvm_host.h int kvm_request_irq_source_id(struct kvm *kvm); · line 1206
Quoted 9 lines
include/linux/kvm_host.h +8
-0+static inline struct kvm_memory_slot *get_slot (struct kvm_memslots *slots,int slot_index) +{ +if (slot_index <0 || slot_index >= slots->used_slots) +return NULL ; + +return &slots->memslots[slot_index]; +} +Since there are plenty of arrays inside struct kvm_memory_slot*, do we want to protect this against speculative out-of-bounds accesses with
array_index_nospec?
I'm not sure when it makes sense to use array_index_nospec. Perhaps this is a good candidate since vcpu->lru_slot_index and the length of memslots[] can both be controlled by userspace.
I'll play around with adding it to see if there are any performance trade-offs.
Quoted 13 lines
include/linux/kvm_host.h search_memslots(struct kvm_memslots *slots, gfn_t gfn) · line 1261
Quoted 8 lines
include/linux/kvm_host.h +6
-0+static inline struct kvm_memory_slot * +search_memslots (struct kvm_memslots *slots,gfn_t gfn) +{ +int slot_index =__search_memslots (slots, gfn); + +return get_slot (slots, slot_index); }Let's use this occasion to remove the duplication between
__gfn_to_memslotandsearch_memslots; you can makesearch_memslotsdo the search and palce the LRU (ehm, MRU) code to__gfn_to_memslotonly. So:
Quoted 7 lines
- the new patch 1 (something like "make
search_memslotssearch without LRU caching") is basically this series's patch 2, plus a tree-wide replacement ofsearch_memslotswith__gfn_to_memslot.- the new patch 2 is this series's patch 1, except
kvm_vcpu_gfn_to_memslotusessearch_memslotsinstead of__search_memslots. The comments in patch 2's commit message about the double misses move to this commit message.
Will do.
Quoted 6 lines
virt/kvm/kvm_main.c EXPORT_SYMBOL_GPL(gfn_to_memslot); · line 2044
Quoted 3 lines
virt/kvm/kvm_main.c +2
-0+if (slot) + vcpu->lru_slot_index = slot_index;Let's call it
lru_slotfor consistency with the field of structkvm_memory_slots.
I'll make sure the two have consistent names when I rename them to get rid of "lru".
Quoted 3 lines
Paolo
Sean Christopherson
maintainerRe: [PATCH 1/6] KVM: Cache the least recently used slot index per vCPU
Quoted 2 lines · David Matlack
I'll include a patch in v2 to fix the name of the existing
lru_slotto something likemru_slotorlast_used_slot.
I vote for last_used_slot, "mru" isn't exactly a common acronym, and the "mr" part could easily be misinterpreted as Memory Region.
PATCH 2/6 KVM: Avoid VM-wide lru_slot lookup in kvm_vcpu_gfn_to_memslot
In reply to David MatlackNow that vCPUs keep track of their own LRU slot, there's no good reason to have them check and update the VM-wide LRU slot. There's no performance data to motivate this change however there are two rationals:
1. Now that vCPUs have their own LRU slot, there's a potential for a double miss (miss the vCPU LRU slot and then miss the VM-wide LRU slot). By avoiding the VM-wide LRU slot check we keep the worst case to a single miss.
2. Large VMs are likely to have multiple memslots and vCPUs accessing different slots. Intuitively, vCPUs will end up thrashing the VM-wide LRU slot, decreasing the LRU hit rate for VM-wide operations such as mmu notifiers and VM ioctls.
--- include/linux/kvm_host.h | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
include/linux/kvm_host.h +11 -9
@@ -1220,17 +1220,13 @@ static inline bool slot_contains_gfn(struct kvm_memslots *slots, int slot_index,static inline int __search_memslots (struct kvm_memslots *slots,gfn_t gfn) {int start =0 , end = slots->used_slots; -int slot =atomic_read (&slots->lru_slot);struct kvm_memory_slot *memslots = slots->memslots;if (unlikely (!slots->used_slots))return -1 ; -if (slot_contains_gfn (slots, slot, gfn)) -return slot; -while (start < end) { - slot = start + (end - start) /2 ; +int slot = start + (end - start) /2 ;if (gfn >= memslots[slot].base_gfn) end = slot;
@@ -1238,10 +1234,8 @@ static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn) start = slot +1 ; } -if (slot_contains_gfn (slots, start, gfn)) { -atomic_set (&slots->lru_slot, start); +if (slot_contains_gfn (slots, start, gfn))return start; - }return -1 ; }
@@ -1255,8 +1249,16 @@ static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn)static inline struct kvm_memory_slot *search_memslots (struct kvm_memslots *slots,gfn_t gfn) { -int slot_index =__search_memslots (slots, gfn); +int slot_index =atomic_read (&slots->lru_slot); + +if (slot_contains_gfn (slots, slot_index, gfn)) +return get_slot (slots, slot_index); + + slot_index =__search_memslots (slots, gfn); +if (slot_index <0 ) +return NULL ; +atomic_set (&slots->lru_slot, slot_index);return get_slot (slots, slot_index); }
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
PATCH 3/6 KVM: x86/mmu: Speed up dirty logging in tdp_mmu_map_handle_target_level
In reply to David MatlackThe existing TDP MMU methods to handle dirty logging are vcpu-agnostic since they can be driven by MMU notifiers and other non-vcpu-specific events in addition to page faults. However this means that the TDP MMU is not benefiting from the new vcpu->lru_slot_index. Fix that by special casing dirty logging in tdp_mmu_map_handle_target_level.
This improves "Populate memory time" in dirty_log_perf_test by 5%:
Command | Before | After ------------------------------- | ---------------- | ------------- ./dirty_log_perf_test -v64 -x64 | 5.472321072s | 5.169832886s
--- arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +12 -3
@@ -929,10 +929,19 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, int write, map_writable, !shadow_accessed_mask, &new_spte); -if (new_spte == iter->old_spte) +if (new_spte == iter->old_spte) { ret =RET_PF_SPURIOUS ; -else if (!tdp_mmu_set_spte_atomic (vcpu->kvm, iter, new_spte)) -return RET_PF_RETRY ; + }else { +if (!tdp_mmu_set_spte_atomic_no_dirty_log (vcpu->kvm, iter, new_spte)) +return RET_PF_RETRY ; + +/* +* Mark the gfn dirty here rather that through the vcpu-agnostic +* handle_changed_spte_dirty_log to leverage vcpu->lru_slot_index. +*/ +if (is_writable_pte (new_spte)) +kvm_vcpu_mark_page_dirty (vcpu, iter->gfn); + }/* * If the page fault was caused by a write but the page is write
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
Paolo Bonzini
maintainerRe: [PATCH 3/6] KVM: x86/mmu: Speed up dirty logging in tdp_mmu_map_handle_target_level
Quoted 6 lines · David Matlack
arch/x86/kvm/mmu/tdp_mmu.c +2
-3-if (new_spte == iter->old_spte) +if (new_spte == iter->old_spte) { ret =RET_PF_SPURIOUS ; -else if (!tdp_mmu_set_spte_atomic (vcpu->kvm, iter, new_spte)) -return RET_PF_RETRY ; + }else {
arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, int write, · line 944
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10
-0+if (!tdp_mmu_set_spte_atomic_no_dirty_log (vcpu->kvm, iter, new_spte)) +return RET_PF_RETRY ; + +/* +* Mark the gfn dirty here rather that through the vcpu-agnostic +* handle_changed_spte_dirty_log to leverage vcpu->lru_slot_index. +*/ +if (is_writable_pte (new_spte)) +kvm_vcpu_mark_page_dirty (vcpu, iter->gfn); + }
Looking at the remaining callers of tdp_mmu_set_spte_atomic we have:
* tdp_mmu_zap_spte_atomic calls it with REMOVED_SPTE as the new_spte, which is never writable
* kvm_tdp_mmu_map calls it for nonleaf SPTEs, which are always writable but should not be dirty.
So I think you should:
* change those two to tdp_mmu_set_spte_atomic_no_dirty_log
* add a WARN_ON_ONCE(iter->level > PG_LEVEL_4K) to tdp_mmu_set_spte_atomic
* put the kvm_vcpu_mark_page_dirty code directly in tdp_mmu_set_spte_atomic, instead of the call to handle_changed_spte_dirty_log
(I can't exclude I'm missing something though).
Paolo
Re: [PATCH 3/6] KVM: x86/mmu: Speed up dirty logging in tdp_mmu_map_handle_target_level
Quoted 29 lines · Paolo Bonzini
Quoted 6 lines · David Matlack
arch/x86/kvm/mmu/tdp_mmu.c +2
-3-if (new_spte == iter->old_spte) +if (new_spte == iter->old_spte) { ret =RET_PF_SPURIOUS ; -else if (!tdp_mmu_set_spte_atomic (vcpu->kvm, iter, new_spte)) -return RET_PF_RETRY ; + }else {arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, int write, · line 944
Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10
-0+if (!tdp_mmu_set_spte_atomic_no_dirty_log (vcpu->kvm, iter, new_spte)) +return RET_PF_RETRY ; + +/* +* Mark the gfn dirty here rather that through the vcpu-agnostic +* handle_changed_spte_dirty_log to leverage vcpu->lru_slot_index. +*/ +if (is_writable_pte (new_spte)) +kvm_vcpu_mark_page_dirty (vcpu, iter->gfn); + }Looking at the remaining callers of
tdp_mmu_set_spte_atomicwe have:*
tdp_mmu_zap_spte_atomiccalls it withREMOVED_SPTEas thenew_spte, which is never writable*
kvm_tdp_mmu_mapcalls it for nonleaf SPTEs, which are always writable but should not be dirty.So I think you should:
Quoted 8 lines
* change those two to
tdp_mmu_set_spte_atomic_no_dirty_log* add a
WARN_ON_ONCE(iter->level > PG_LEVEL_4K)totdp_mmu_set_spte_atomic* put the
kvm_vcpu_mark_page_dirtycode directly intdp_mmu_set_spte_atomic, instead of the call tohandle_changed_spte_dirty_log(I can't exclude I'm missing something though).
Makes sense. I'll take a look to confirm and make those changes in v2.
Quoted 3 lines
Paolo
PATCH 4/6 KVM: x86/mmu: Leverage vcpu->lru_slot_index for rmap_add and rmap_recycle
In reply to David Matlackrmap_add() and rmap_recycle() both run in the context of the vCPU and thus we can use kvm_vcpu_gfn_to_memslot() to look up the memslot. This enables rmap_add() and rmap_recycle() to take advantage of vcpu->lru_slot_index and avoid expensive memslot searching.
This change improves the performance of "Populate memory time" in dirty_log_perf_test with tdp_mmu=N. In addition to improving the performance, "Populate memory time" no longer scales with the number of memslots in the VM.
Command | Before | After ------------------------------- | ---------------- | ------------- ./dirty_log_perf_test -v64 -x1 | 15.18001570s | 14.99469366s ./dirty_log_perf_test -v64 -x64 | 18.71336392s | 14.98675076s
--- arch/x86/kvm/mmu/mmu.c | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/mmu.c +20 -15
@@ -1043,17 +1043,6 @@ static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,return &slot->arch.rmap[level -PG_LEVEL_4K ][idx]; } -static struct kvm_rmap_head *gfn_to_rmap (struct kvm *kvm,gfn_t gfn, -struct kvm_mmu_page *sp) -{ -struct kvm_memslots *slots; -struct kvm_memory_slot *slot; - - slots =kvm_memslots_for_spte_role (kvm, sp->role); - slot =__gfn_to_memslot (slots, gfn); -return __gfn_to_rmap (gfn, sp->role.level, slot); -} -static bool rmap_can_add (struct kvm_vcpu *vcpu) {struct kvm_mmu_memory_cache *mc;
@@ -1064,24 +1053,39 @@ static bool rmap_can_add(struct kvm_vcpu *vcpu)static int rmap_add (struct kvm_vcpu *vcpu,u64 *spte,gfn_t gfn) { +struct kvm_memory_slot *slot;struct kvm_mmu_page *sp;struct kvm_rmap_head *rmap_head; sp =sptep_to_sp (spte);kvm_mmu_page_set_gfn (sp, spte - sp->spt, gfn); - rmap_head =gfn_to_rmap (vcpu->kvm, gfn, sp); + slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot);return pte_list_add (vcpu, spte, rmap_head); } +static void rmap_remove (struct kvm *kvm,u64 *spte) { +struct kvm_memslots *slots; +struct kvm_memory_slot *slot;struct kvm_mmu_page *sp;gfn_t gfn;struct kvm_rmap_head *rmap_head; sp =sptep_to_sp (spte); gfn =kvm_mmu_page_get_gfn (sp, spte - sp->spt); - rmap_head =gfn_to_rmap (kvm, gfn, sp); + +/* +* Unlike rmap_add and rmap_recycle, rmap_remove does not run in the +* context of a vCPU so have to determine which memslots to use based +* on context information in sp->role. +*/ + slots =kvm_memslots_for_spte_role (kvm, sp->role); + + slot =__gfn_to_memslot (slots, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); +__pte_list_remove (spte, rmap_head); }
@@ -1628,12 +1632,13 @@ static bool kvm_test_age_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,static void rmap_recycle (struct kvm_vcpu *vcpu,u64 *spte,gfn_t gfn) { +struct kvm_memory_slot *slot;struct kvm_rmap_head *rmap_head;struct kvm_mmu_page *sp; sp =sptep_to_sp (spte); - - rmap_head =gfn_to_rmap (vcpu->kvm, gfn, sp); + slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot);kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
Re: [PATCH 4/6] KVM: x86/mmu: Leverage vcpu->lru_slot_index for rmap_add and rmap_recycle
Quoted 10 lines · David Matlack
rmap_add()andrmap_recycle()both run in the context of the vCPU and thus we can usekvm_vcpu_gfn_to_memslot()to look up the memslot. This enablesrmap_add()andrmap_recycle()to take advantage ofvcpu->lru_slot_indexand avoid expensive memslot searching.This change improves the performance of "Populate memory time" in
dirty_log_perf_testwith tdp_mmu=N. In addition to improving the performance, "Populate memory time" no longer scales with the number of memslots in the VM.
Quoted 6 lines
Command | Before | After ------------------------------- | ---------------- | ------------- ./dirty_log_perf_test -v64 -x1 | 15.18001570s | 14.99469366s ./dirty_log_perf_test -v64 -x64 | 18.71336392s | 14.98675076sSigned-off-by David Matlack <[email protected]>
Quoted 85 lines
--- arch/x86/kvm/mmu/mmu.c | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-)arch/x86/kvm/mmu/mmu.c +20
-15@@ -1043,17 +1043,6 @@ static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,return &slot->arch.rmap[level -PG_LEVEL_4K ][idx]; } -static struct kvm_rmap_head *gfn_to_rmap (struct kvm *kvm,gfn_t gfn, -struct kvm_mmu_page *sp) -{ -struct kvm_memslots *slots; -struct kvm_memory_slot *slot; - - slots =kvm_memslots_for_spte_role (kvm, sp->role); - slot =__gfn_to_memslot (slots, gfn); -return __gfn_to_rmap (gfn, sp->role.level, slot); -} -static bool rmap_can_add (struct kvm_vcpu *vcpu) {struct kvm_mmu_memory_cache *mc;@@ -1064,24 +1053,39 @@ static bool rmap_can_add(struct kvm_vcpu *vcpu)static int rmap_add (struct kvm_vcpu *vcpu,u64 *spte,gfn_t gfn) { +struct kvm_memory_slot *slot;struct kvm_mmu_page *sp;struct kvm_rmap_head *rmap_head; sp =sptep_to_sp (spte);kvm_mmu_page_set_gfn (sp, spte - sp->spt, gfn); - rmap_head =gfn_to_rmap (vcpu->kvm, gfn, sp); + slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot);return pte_list_add (vcpu, spte, rmap_head); } +static void rmap_remove (struct kvm *kvm,u64 *spte) { +struct kvm_memslots *slots; +struct kvm_memory_slot *slot;struct kvm_mmu_page *sp;gfn_t gfn;struct kvm_rmap_head *rmap_head; sp =sptep_to_sp (spte); gfn =kvm_mmu_page_get_gfn (sp, spte - sp->spt); - rmap_head =gfn_to_rmap (kvm, gfn, sp); + +/* +* Unlike rmap_add and rmap_recycle, rmap_remove does not run in the +* context of a vCPU so have to determine which memslots to use based +* on context information in sp->role. +*/ + slots =kvm_memslots_for_spte_role (kvm, sp->role); + + slot =__gfn_to_memslot (slots, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); +__pte_list_remove (spte, rmap_head); }@@ -1628,12 +1632,13 @@ static bool kvm_test_age_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,static void rmap_recycle (struct kvm_vcpu *vcpu,u64 *spte,gfn_t gfn) { +struct kvm_memory_slot *slot;struct kvm_rmap_head *rmap_head;struct kvm_mmu_page *sp; sp =sptep_to_sp (spte); - - rmap_head =gfn_to_rmap (vcpu->kvm, gfn, sp); + slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); + rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot);kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
PATCH 5/6 KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap
In reply to David Matlackgfn_to_rmap was removed in the previous patch so there is no need to retain the double underscore on __gfn_to_rmap.
--- arch/x86/kvm/mmu/mmu.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -13
@@ -1034,8 +1034,8 @@ static bool pte_list_destroy(struct kvm_rmap_head *rmap_head,return true; } -static struct kvm_rmap_head *__gfn_to_rmap (gfn_t gfn,int level, -const struct kvm_memory_slot *slot) +static struct kvm_rmap_head *gfn_to_rmap (gfn_t gfn,int level, +const struct kvm_memory_slot *slot) {unsigned long idx;
@@ -1060,7 +1060,7 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp =sptep_to_sp (spte);kvm_mmu_page_set_gfn (sp, spte - sp->spt, gfn); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);return pte_list_add (vcpu, spte, rmap_head); }
@@ -1084,7 +1084,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) slots =kvm_memslots_for_spte_role (kvm, sp->role); slot =__gfn_to_memslot (slots, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);__pte_list_remove (spte, rmap_head); }
@@ -1306,8 +1306,8 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,return ;while (mask) { - rmap_head =__gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), -PG_LEVEL_4K , slot); + rmap_head =gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), +PG_LEVEL_4K , slot);__rmap_write_protect (kvm, rmap_head, false);/* clear the first set bit */
@@ -1339,8 +1339,8 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,return ;while (mask) { - rmap_head =__gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), -PG_LEVEL_4K , slot); + rmap_head =gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), +PG_LEVEL_4K , slot);__rmap_clear_dirty (kvm, rmap_head, slot);/* clear the first set bit */
@@ -1406,7 +1406,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,if (kvm_memslots_have_rmaps (kvm)) {for (i = min_level; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { - rmap_head =__gfn_to_rmap (gfn, i, slot); + rmap_head =gfn_to_rmap (gfn, i, slot); write_protected |=__rmap_write_protect (kvm, rmap_head, true); } }
@@ -1506,9 +1506,8 @@ rmap_walk_init_level(struct slot_rmap_walk_iterator *iterator, int level) { iterator->level = level; iterator->gfn = iterator->start_gfn; - iterator->rmap =__gfn_to_rmap (iterator->gfn, level, iterator->slot); - iterator->end_rmap =__gfn_to_rmap (iterator->end_gfn, level, - iterator->slot); + iterator->rmap =gfn_to_rmap (iterator->gfn, level, iterator->slot); + iterator->end_rmap =gfn_to_rmap (iterator->end_gfn, level, iterator->slot); } static void
@@ -1638,7 +1637,7 @@ static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp =sptep_to_sp (spte); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
kernel test robot
build botRe: [PATCH 5/6] KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap
Thank you for the patch! Yet something to improve:
Hi David,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[cannot apply to vhost/linux-next v5.14-rc3 next-20210730] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/David-Matlack/Improve-gfn-to-memslot-performance-during-page-faults/20210731-063919 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue config: i386-randconfig-a012-20210730 (attached as .config) compiler: gcc-10 (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0 reproduce (this is a W=1 build): # https://github.com/0day-ci/linux/commit/0310eccd630f37e334f797d966bb515ab3c3b3d2 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review David-Matlack/Improve-gfn-to-memslot-performance-during-page-faults/20210731-063919 git checkout 0310eccd630f37e334f797d966bb515ab3c3b3d2 # save the attached .config to linux build tree make W=1 ARCH=i386
If you fix the issue, kindly add following tag as appropriate
All errors (new ones prefixed by >>):
In file included from arch/x86/kvm/mmu/mmu.c:1936:
arch/x86/kvm/mmu/mmu_audit.c: In function 'inspect_spte_has_rmap':
arch/x86/kvm/mmu/mmu_audit.c:150:14: error: implicit declaration of function '__gfn_to_rmap'; did you mean 'gfn_to_rmap'? [-Werror=implicit-function-declaration]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^~~~~~~~~~~~~
| gfn_to_rmap
>> arch/x86/kvm/mmu/mmu_audit.c:150:12: error: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^
arch/x86/kvm/mmu/mmu_audit.c: In function 'audit_write_protection':
arch/x86/kvm/mmu/mmu_audit.c:203:12: error: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
203 | rmap_head = __gfn_to_rmap(sp->gfn, PG_LEVEL_4K, slot);
| ^cc1: all warnings being treated as errors
vim +150 arch/x86/kvm/mmu/mmu_audit.c
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 125
eb2591865a234c6f arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 126 static void inspect_spte_has_rmap(struct kvm *kvm, u64 *sptep)
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 127 {
bd80158aff71a802 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 128 static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 10);
018aabb56d6109c8 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 129 struct kvm_rmap_head *rmap_head;
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 130 struct kvm_mmu_page *rev_sp;
699023e239658e62 arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 131 struct kvm_memslots *slots;
699023e239658e62 arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 132 struct kvm_memory_slot *slot;
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 133 gfn_t gfn;
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 134
573546820b792ef6 arch/x86/kvm/mmu/mmu_audit.c Sean Christopherson 2020-06-22 135 rev_sp = sptep_to_sp(sptep);
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 136 gfn = kvm_mmu_page_get_gfn(rev_sp, sptep - rev_sp->spt);
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 137
699023e239658e62 arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 138 slots = kvm_memslots_for_spte_role(kvm, rev_sp->role);
699023e239658e62 arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 139 slot = __gfn_to_memslot(slots, gfn);
699023e239658e62 arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 140 if (!slot) {
bd80158aff71a802 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 141 if (!__ratelimit(&ratelimit_state))
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 142 return;
b034cf0105235e65 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 143 audit_printk(kvm, "no memslot for gfn %llx\n", gfn);
b034cf0105235e65 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 144 audit_printk(kvm, "index %ld of sp (gfn=%llx)\n",
38904e128778c388 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-09-27 145 (long int)(sptep - rev_sp->spt), rev_sp->gfn);
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 146 dump_stack();
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 147 return;
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 148 }
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 149
018aabb56d6109c8 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 @150 rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
018aabb56d6109c8 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 151 if (!rmap_head->val) {
bd80158aff71a802 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 152 if (!__ratelimit(&ratelimit_state))
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 153 return;
b034cf0105235e65 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 154 audit_printk(kvm, "no rmap for writable spte %llx\n",
b034cf0105235e65 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 155 *sptep);
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 156 dump_stack();
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 157 }
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 158 }
2f4f337248cd5660 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 159 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
kernel test robot
build botRe: [PATCH 5/6] KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap
In reply to David MatlackThank you for the patch! Yet something to improve:
Hi David,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[cannot apply to vhost/linux-next v5.14-rc3 next-20210730] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/David-Matlack/Improve-gfn-to-memslot-performance-during-page-faults/20210731-063919 base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue config: x86_64-rhel-8.3-kselftests (attached as .config) compiler: gcc-10 (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0 reproduce: # apt-get install sparse # sparse version: v0.6.3-341-g8af24329-dirty # https://github.com/0day-ci/linux/commit/0310eccd630f37e334f797d966bb515ab3c3b3d2 git remote add linux-review https://github.com/0day-ci/linux git fetch --no-tags linux-review David-Matlack/Improve-gfn-to-memslot-performance-during-page-faults/20210731-063919 git checkout 0310eccd630f37e334f797d966bb515ab3c3b3d2 # save the attached .config to linux build tree make W=1 C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
All error/warnings (new ones prefixed by >>):
In file included from arch/x86/kvm/mmu/mmu.c:1936:
arch/x86/kvm/mmu/mmu_audit.c: In function 'inspect_spte_has_rmap':
>> arch/x86/kvm/mmu/mmu_audit.c:150:14: error: implicit declaration of function '__gfn_to_rmap'; did you mean 'gfn_to_rmap'? [-Werror=implicit-function-declaration]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^~~~~~~~~~~~~
| gfn_to_rmap
>> arch/x86/kvm/mmu/mmu_audit.c:150:12: warning: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^
arch/x86/kvm/mmu/mmu_audit.c: In function 'audit_write_protection':
arch/x86/kvm/mmu/mmu_audit.c:203:12: warning: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
203 | rmap_head = __gfn_to_rmap(sp->gfn, PG_LEVEL_4K, slot);
| ^ cc1: some warnings being treated as errors
--
In file included from arch/x86/kvm/mmu/mmu.c:1936:
arch/x86/kvm/mmu/mmu_audit.c: In function 'inspect_spte_has_rmap':
>> arch/x86/kvm/mmu/mmu_audit.c:150:14: error: implicit declaration of function '__gfn_to_rmap'; did you mean 'gfn_to_rmap'? [-Werror=implicit-function-declaration]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^~~~~~~~~~~~~
| gfn_to_rmap
>> arch/x86/kvm/mmu/mmu_audit.c:150:12: warning: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
150 | rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
| ^
arch/x86/kvm/mmu/mmu_audit.c: In function 'audit_write_protection':
arch/x86/kvm/mmu/mmu_audit.c:203:12: warning: assignment to 'struct kvm_rmap_head *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
203 | rmap_head = __gfn_to_rmap(sp->gfn, PG_LEVEL_4K, slot);
| ^
cc1: some warnings being treated as errors
make[3]: *** [scripts/Makefile.build:271: arch/x86/kvm/mmu/mmu.o] Error 1
make[3]: Target '__build' not remade because of errors.
make[2]: *** [scripts/Makefile.build:514: arch/x86/kvm] Error 2
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1842: arch/x86] Error 2
make[1]: Target 'vmlinux' not remade because of errors.
make: *** [Makefile:220: __sub-make] Error 2
make: Target 'vmlinux' not remade because of errors.
***
*** The source tree is not clean, please run 'make ARCH=x86_64 mrproper'
*** in /kbuild/worktree/build-ktools-consumer
***
make[2]: *** [Makefile:547: outputmakefile] Error 1
make[2]: Target 'syncconfig' not remade because of errors.
make[1]: *** [Makefile:710: include/config/auto.conf.cmd] Error 2
make[1]: *** [include/config/auto.conf.cmd] Deleting file 'include/generated/autoconf.h'
make[1]: Failed to remake makefile 'include/config/auto.conf.cmd'.
make[1]: Failed to remake makefile 'include/config/auto.conf'.
***
*** The source tree is not clean, please run 'make ARCH=x86_64 mrproper'
*** in /kbuild/worktree/build-ktools-consumer
***
make[1]: *** [Makefile:547: outputmakefile] Error 1
In file included from <command-line>:32:
include/linux/kconfig.h:7:10: fatal error: generated/autoconf.h: No such file or directory
7 | #include <generated/autoconf.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from <command-line>:32:
include/linux/kconfig.h:7:10: fatal error: generated/autoconf.h: No such file or directory
7 | #include <generated/autoconf.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from <command-line>:32:
include/linux/kconfig.h:7:10: fatal error: generated/autoconf.h: No such file or directory
7 | #include <generated/autoconf.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from scripts/selinux/mdp/mdp.c:22:
include/linux/kconfig.h:7:10: fatal error: generated/autoconf.h: No such file or directory
7 | #include <generated/autoconf.h>
| ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [scripts/Makefile.host:95: scripts/selinux/mdp/mdp] Error 1
make[4]: Target '__build' not remade because of errors.
make[3]: *** [scripts/Makefile.build:496: scripts/selinux/mdp] Error 2
make[3]: Target '__build' not remade because of errors.
make[2]: *** [scripts/Makefile.build:496: scripts/selinux] Error 2
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1190: scripts] Error 2
In file included from tools/include/uapi/linux/ethtool.h:19,
from xsk.c:18:
usr/include/linux/if_ether.h:169:1: error: packed attribute is unnecessary for 'ethhdr' [-Werror=packed]
169 | } __attribute__((packed));
| ^ cc1: all warnings being treated as errors
make[5]: *** [tools/build/Makefile.build:96: tools/bpf/resolve_btfids/staticobjs/xsk.o] Error 1 make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile:179: tools/bpf/resolve_btfids/staticobjs/libbpf-in.o] Error 2 make[3]: *** [Makefile:56: tools/bpf/resolve_btfids//libbpf.a] Error 2
make[2]: *** [Makefile:71: bpf/resolve_btfids] Error 2
make[1]: *** [Makefile:1930: tools/bpf/resolve_btfids] Error 2
make[1]: Target 'vmlinux' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2
make: Target 'vmlinux' not remade because of errors.
vim +150 arch/x86/kvm/mmu/mmu_audit.c
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 125
eb2591865a234c arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 126 static void inspect_spte_has_rmap(struct kvm *kvm, u64 *sptep)
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 127 {
bd80158aff71a8 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 128 static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 10);
018aabb56d6109 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 129 struct kvm_rmap_head *rmap_head;
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 130 struct kvm_mmu_page *rev_sp;
699023e239658e arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 131 struct kvm_memslots *slots;
699023e239658e arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 132 struct kvm_memory_slot *slot;
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 133 gfn_t gfn;
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 134
573546820b792e arch/x86/kvm/mmu/mmu_audit.c Sean Christopherson 2020-06-22 135 rev_sp = sptep_to_sp(sptep);
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 136 gfn = kvm_mmu_page_get_gfn(rev_sp, sptep - rev_sp->spt);
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 137
699023e239658e arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 138 slots = kvm_memslots_for_spte_role(kvm, rev_sp->role);
699023e239658e arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 139 slot = __gfn_to_memslot(slots, gfn);
699023e239658e arch/x86/kvm/mmu_audit.c Paolo Bonzini 2015-05-18 140 if (!slot) {
bd80158aff71a8 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 141 if (!__ratelimit(&ratelimit_state))
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 142 return;
b034cf0105235e arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 143 audit_printk(kvm, "no memslot for gfn %llx\n", gfn);
b034cf0105235e arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 144 audit_printk(kvm, "index %ld of sp (gfn=%llx)\n",
38904e128778c3 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-09-27 145 (long int)(sptep - rev_sp->spt), rev_sp->gfn);
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 146 dump_stack();
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 147 return;
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 148 }
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 149
018aabb56d6109 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 @150 rmap_head = __gfn_to_rmap(gfn, rev_sp->role.level, slot);
018aabb56d6109 arch/x86/kvm/mmu_audit.c Takuya Yoshikawa 2015-11-20 151 if (!rmap_head->val) {
bd80158aff71a8 arch/x86/kvm/mmu_audit.c Jan Kiszka 2011-09-12 152 if (!__ratelimit(&ratelimit_state))
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 153 return;
b034cf0105235e arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 154 audit_printk(kvm, "no rmap for writable spte %llx\n",
b034cf0105235e arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-12-23 155 *sptep);
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 156 dump_stack();
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 157 }
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 158 }
2f4f337248cd56 arch/x86/kvm/mmu_audit.c Xiao Guangrong 2010-08-30 159 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Re: [PATCH 5/6] KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap
In reply to David MatlackQuoted 7 lines · David Matlack
gfn_to_rmapwas removed in the previous patch so there is no need to retain the double underscore on__gfn_to_rmap.Signed-off-by David Matlack <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-)
apart from the extra mmu_audit.c changes.
Paolo
Quoted 86 lines
arch/x86/kvm/mmu/mmu.c +12
-13@@ -1034,8 +1034,8 @@ static bool pte_list_destroy(struct kvm_rmap_head *rmap_head,return true; } -static struct kvm_rmap_head *__gfn_to_rmap (gfn_t gfn,int level, -const struct kvm_memory_slot *slot) +static struct kvm_rmap_head *gfn_to_rmap (gfn_t gfn,int level, +const struct kvm_memory_slot *slot) {unsigned long idx;@@ -1060,7 +1060,7 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp =sptep_to_sp (spte);kvm_mmu_page_set_gfn (sp, spte - sp->spt, gfn); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);return pte_list_add (vcpu, spte, rmap_head); }@@ -1084,7 +1084,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) slots =kvm_memslots_for_spte_role (kvm, sp->role); slot =__gfn_to_memslot (slots, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);__pte_list_remove (spte, rmap_head); }@@ -1306,8 +1306,8 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,return ;while (mask) { - rmap_head =__gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), -PG_LEVEL_4K , slot); + rmap_head =gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), +PG_LEVEL_4K , slot);__rmap_write_protect (kvm, rmap_head, false);/* clear the first set bit */ @@ -1339,8 +1339,8 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,return ;while (mask) { - rmap_head =__gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), -PG_LEVEL_4K , slot); + rmap_head =gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask), +PG_LEVEL_4K , slot);__rmap_clear_dirty (kvm, rmap_head, slot);/* clear the first set bit */ @@ -1406,7 +1406,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,if (kvm_memslots_have_rmaps (kvm)) {for (i = min_level; i <=KVM_MAX_HUGEPAGE_LEVEL ; ++i) { - rmap_head =__gfn_to_rmap (gfn, i, slot); + rmap_head =gfn_to_rmap (gfn, i, slot); write_protected |=__rmap_write_protect (kvm, rmap_head, true); } }@@ -1506,9 +1506,8 @@ rmap_walk_init_level(struct slot_rmap_walk_iterator *iterator, int level) { iterator->level = level; iterator->gfn = iterator->start_gfn; - iterator->rmap =__gfn_to_rmap (iterator->gfn, level, iterator->slot); - iterator->end_rmap =__gfn_to_rmap (iterator->end_gfn, level, - iterator->slot); + iterator->rmap =gfn_to_rmap (iterator->gfn, level, iterator->slot); + iterator->end_rmap =gfn_to_rmap (iterator->end_gfn, level, iterator->slot); } static void@@ -1638,7 +1637,7 @@ static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp =sptep_to_sp (spte); slot =kvm_vcpu_gfn_to_memslot (vcpu, gfn); - rmap_head =__gfn_to_rmap (gfn, sp->role.level, slot); + rmap_head =gfn_to_rmap (gfn, sp->role.level, slot);kvm_unmap_rmapp (vcpu->kvm, rmap_head,NULL , gfn, sp->role.level,__pte (0 )); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn,
PATCH 6/6 KVM: selftests: Support multiple slots in dirty_log_perf_test
In reply to David MatlackIntroduce a new option to dirty_log_perf_test: -x number_of_slots. This causes the test to attempt to split the region of memory into the given number of slots. If the region cannot be evenly divided, the test will fail.
This allows testing with more than one slot and therefore measure how performance scales with the number of memslots.
--- .../selftests/kvm/access_tracking_perf_test.c | 2 +- .../selftests/kvm/demand_paging_test.c | 2 +- .../selftests/kvm/dirty_log_perf_test.c | 76 ++++++++++++++++--- .../selftests/kvm/include/perf_test_util.h | 2 +- .../selftests/kvm/lib/perf_test_util.c | 20 +++-- .../kvm/memslot_modification_stress_test.c | 2 +- 6 files changed, 84 insertions(+), 20 deletions(-)
tools/testing/selftests/kvm/access_tracking_perf_test.c +1 -1
@@ -333,7 +333,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)pthread_t *vcpu_threads;int vcpus = params->vcpus; - vm =perf_test_create_vm (mode, vcpus, params->vcpu_memory_bytes, + vm =perf_test_create_vm (mode, vcpus, params->vcpu_memory_bytes,1 , params->backing_src); perf_test_setup_vcpus(vm, vcpus, params->vcpu_memory_bytes,
tools/testing/selftests/kvm/demand_paging_test.c +1 -1
@@ -293,7 +293,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)int vcpu_id;int r; - vm =perf_test_create_vm (mode, nr_vcpus, guest_percpu_mem_size, + vm =perf_test_create_vm (mode, nr_vcpus, guest_percpu_mem_size,1 , p->src_type); perf_test_args.wr_fract =1 ;
tools/testing/selftests/kvm/dirty_log_perf_test.c +65 -11
144 lines not on this page · raw message
tools/testing/selftests/kvm/include/perf_test_util.h +1 -1
@@ -44,7 +44,7 @@ extern struct perf_test_args perf_test_args;extern uint64_t guest_test_phys_mem;struct kvm_vm *perf_test_create_vm (enum vm_guest_mode mode,int vcpus, -uint64_t vcpu_memory_bytes, +uint64_t vcpu_memory_bytes,int slots,enum vm_mem_backing_src_type backing_src);void perf_test_destroy_vm (struct kvm_vm *vm); void perf_test_setup_vcpus(struct kvm_vm *vm, int vcpus,
tools/testing/selftests/kvm/lib/perf_test_util.c +15 -5
@@ -50,11 +50,12 @@ static void guest_code(uint32_t vcpu_id) }struct kvm_vm *perf_test_create_vm (enum vm_guest_mode mode,int vcpus, -uint64_t vcpu_memory_bytes, +uint64_t vcpu_memory_bytes,int slots,enum vm_mem_backing_src_type backing_src) {struct kvm_vm *vm;uint64_t guest_num_pages; +int i;pr_info ("Testing guest mode: %s\n" ,vm_guest_mode_string (mode));
@@ -68,6 +69,9 @@ struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, "Guest memory size is not host page size aligned.");TEST_ASSERT (vcpu_memory_bytes % perf_test_args.guest_page_size ==0 ,"Guest memory size is not guest page size aligned." ); +TEST_ASSERT (guest_num_pages % slots ==0 , +"Guest memory cannot be evenly divided into %d slots." , + slots); vm = vm_create_with_vcpus(mode, vcpus, DEFAULT_GUEST_PHY_PAGES, (vcpus * vcpu_memory_bytes) / perf_test_args.guest_page_size,
@@ -95,10 +99,16 @@ struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus,#endif pr_info ("guest physical test memory offset: 0x%lx\n" , guest_test_phys_mem); -/* Add an extra memory slot for testing */ -vm_userspace_mem_region_add (vm, backing_src, guest_test_phys_mem, -PERF_TEST_MEM_SLOT_INDEX , - guest_num_pages,0 ); +/* Add extra memory slots for testing */ +for (i =0 ; i < slots; i++) { +uint64_t region_pages = guest_num_pages / slots; +vm_paddr_t region_start = guest_test_phys_mem + + region_pages * perf_test_args.guest_page_size * i; + +vm_userspace_mem_region_add (vm, backing_src, region_start, +PERF_TEST_MEM_SLOT_INDEX + i, + region_pages,0 ); + }/* Do mapping for the demand paging memory slot */ virt_map (vm, guest_test_virt_mem, guest_test_phys_mem, guest_num_pages);
tools/testing/selftests/kvm/memslot_modification_stress_test.c +1 -1
@@ -105,7 +105,7 @@ static void run_test(enum vm_guest_mode mode, void *arg)struct kvm_vm *vm;int vcpu_id; - vm =perf_test_create_vm (mode, nr_vcpus, guest_percpu_mem_size, + vm =perf_test_create_vm (mode, nr_vcpus, guest_percpu_mem_size,1 ,VM_MEM_SRC_ANONYMOUS ); perf_test_args.wr_fract =1 ;
Signature · 1 line
2.32.0.554.ge1b32706d8-goog