KVM: x86/mmu: Dirty logging fixes and improvements
Sean Christopherson
maintainercover KVM: x86/mmu: Dirty logging fixes and improvements
Paolo, this is more or less ready, but on final read-through before sending I realized it would be a good idea to WARN during VM destruction if cpu_dirty_logging_count is non-zero. I wanted to get you this before the 5.12 window opens in case you want the TDP MMU fixes for 5.12. I'll do the above change and retest next week (note, Monday is a US holiday).
On to the code...
This started out as a small tweak to collapsible SPTE zapping in the TDP MMU, and ended up as a rather large overhaul of CPU dirty logging, a.k.a. PML.
Four main highlights:
- Do a more precise check on whether or not a SPTE should be zapped to rebuild it as a large page.
- Disable PML when running L2. PML is fully emulated for L1 VMMs, thus enabling PML in L2 can only hurt and never help.
- Drop the existing PML kvm_x86_ops. They're basically trampolines into the MMU, and IMO do far more harm than good.
- Turn on PML only when it's needed instead of setting all dirty bits to soft disable PML.
What led me down the rabbit's hole of ripping out the existing PML kvm_x86_ops isn't really shown here. Prior to incorporating Makarand's patch, which allowed for the wholesale remove of setting dirty bits, I spent a bunch of time poking around the "set dirty bits" code. My original changes optimized that path to skip setting dirty bits in the nested MMU, since the nested MMU relies on write-protection and not PML. That in turn allowed the TDP MMU zapping to completely skip walking the rmaps, but doing so based on a bunch of callbacks was a twisted mess.
Happily, those patches got dropped in favor of nuking the code entirely.
Ran selftest and unit tests, and migrated actual VMs on AMD and Intel, with and without TDP MMU, and with and without EPT. The AMD system I'm testing on infinite loops on the reset vector due to a #PF when NPT is disabled, so that didn't get tested. That reproduces with kvm/next, I'll dig into it next week (no idea if it's a KVM or hardware issue).
For actual migration, I ran kvm-unit-tests in L1 along with stress to hammer memory, and verified migration was effectively blocked until the stress threads were killed (I didn't feel like figuring out how to throttle the VM).
Makarand Sonare (1):
KVM: VMX: Dynamically enable/disable PML based on memslot dirty
logging
Sean Christopherson (13):
KVM: x86/mmu: Expand collapsible SPTE zap for TDP MMU to ZONE_DEVICE
pages
KVM: x86/mmu: Don't unnecessarily write-protect small pages in TDP MMU
KVM: x86/mmu: Split out max mapping level calculation to helper
KVM: x86/mmu: Pass the memslot to the rmap callbacks
KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
KVM: nVMX: Disable PML in hardware when running L2
KVM: x86/mmu: Expand on the comment in
kvm_vcpu_ad_need_write_protect()
KVM: x86/mmu: Make dirty log size hook (PML) a value, not a function
KVM: x86: Move MMU's PML logic to common code
KVM: x86: Further clarify the logic and comments for toggling log
dirty
KVM: x86/mmu: Don't set dirty bits when disabling dirty logging w/ PML
KVM: x86: Fold "write-protect large" use case into generic
write-protect
KVM: x86/mmu: Remove a variety of unnecessary exports
arch/x86/include/asm/kvm-x86-ops.h | 6 +- arch/x86/include/asm/kvm_host.h | 36 +---- arch/x86/kvm/mmu/mmu.c | 203 +++++++++-------------------- arch/x86/kvm/mmu/mmu_internal.h | 7 +- arch/x86/kvm/mmu/tdp_mmu.c | 66 +--------- arch/x86/kvm/mmu/tdp_mmu.h | 3 +- arch/x86/kvm/vmx/nested.c | 34 +++-- arch/x86/kvm/vmx/vmx.c | 94 +++++-------- arch/x86/kvm/vmx/vmx.h | 2 + arch/x86/kvm/x86.c | 145 +++++++++++++-------- 10 files changed, 230 insertions(+), 366 deletions(-)
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 1/14 KVM: x86/mmu: Expand collapsible SPTE zap for TDP MMU to ZONE_DEVICE pages
Zap SPTEs that are backed by ZONE_DEVICE pages when zappings SPTEs to rebuild them as huge pages in the TDP MMU. ZONE_DEVICE huge pages are managed differently than "regular" pages and are not compound pages.
--- arch/x86/kvm/mmu/tdp_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/tdp_mmu.c +2 -1
@@ -1348,7 +1348,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - !PageTransCompoundMap (pfn_to_page (pfn))) + (!PageTransCompoundMap (pfn_to_page (pfn)) && + !kvm_is_zone_device_pfn (pfn)))continue ;tdp_mmu_set_spte (kvm, &iter,0 );
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Paolo Bonzini
maintainerRe: [PATCH 01/14] KVM: x86/mmu: Expand collapsible SPTE zap for TDP MMU to ZONE_DEVICE pages
Quoted 17 lines · Sean Christopherson
Zap SPTEs that are backed by
ZONE_DEVICEpages when zappings SPTEs to rebuild them as huge pages in the TDP MMU.ZONE_DEVICEhuge pages are managed differently than "regular" pages and are not compound pages.Cc Ben Gardon <[email protected]>Fixes 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU")Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/tdp_mmu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)arch/x86/kvm/mmu/tdp_mmu.c +0
-0
arch/x86/kvm/mmu/tdp_mmu.c static void zap_collapsible_spte_range(struct kvm *kvm, · line 1355
Quoted 9 lines
arch/x86/kvm/mmu/tdp_mmu.c +2
-1@@ -1348,7 +1348,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - !PageTransCompoundMap (pfn_to_page (pfn))) + (!PageTransCompoundMap (pfn_to_page (pfn)) && + !kvm_is_zone_device_pfn (pfn)))continue ;tdp_mmu_set_spte (kvm, &iter,0 );
I added a note to the commit message that a similar check is found in kvm_mmu_zap_collapsible_spte.
Paolo
Sean Christopherson
maintainerPATCH 2/14 KVM: x86/mmu: Don't unnecessarily write-protect small pages in TDP MMU
In reply to Sean ChristophersonRespect start_level when write-protect pages in the TDP MMU for dirty logging. When the dirty bitmaps are initialized with all bits set, small pages don't need to be write-protected as they've already been marked dirty.
--- arch/x86/kvm/mmu/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -5500,7 +5500,7 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, flush =slot_handle_level (kvm, memslot, slot_rmap_write_protect, start_level,KVM_MAX_HUGEPAGE_LEVEL , false);if (is_tdp_mmu_enabled (kvm)) - flush |=kvm_tdp_mmu_wrprot_slot (kvm, memslot,PG_LEVEL_4K ); + flush |=kvm_tdp_mmu_wrprot_slot (kvm, memslot, start_level);write_unlock (&kvm->mmu_lock);/*
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 3/14 KVM: x86/mmu: Split out max mapping level calculation to helper
In reply to Sean ChristophersonFactor out the logic for determining the maximum mapping level given a memslot and a gpa. The helper will be used when zapping collapsible SPTEs when disabling dirty logging, e.g. to avoid zapping SPTEs that can't possibly be rebuilt as hugepages.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 37 ++++++++++++++++++++------------- arch/x86/kvm/mmu/mmu_internal.h | 2 ++ 2 files changed, 24 insertions(+), 15 deletions(-)
arch/x86/kvm/mmu/mmu.c +22 -15
@@ -2756,8 +2756,8 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep)__direct_pte_prefetch (vcpu, sp, sptep); } -static int host_pfn_mapping_level (struct kvm_vcpu *vcpu,gfn_t gfn, -kvm_pfn_t pfn,struct kvm_memory_slot *slot) +static int host_pfn_mapping_level (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +struct kvm_memory_slot *slot) {unsigned long hva;pte_t *pte;
@@ -2776,19 +2776,36 @@ static int host_pfn_mapping_level(struct kvm_vcpu *vcpu, gfn_t gfn,*/ hva =__gfn_to_hva_memslot (slot, gfn); - pte =lookup_address_in_mm (vcpu->kvm->mm, hva, &level); + pte =lookup_address_in_mm (kvm->mm, hva, &level);if (unlikely (!pte))return PG_LEVEL_4K ;return level; } +int kvm_mmu_max_mapping_level (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t pfn,int max_level) +{ +struct kvm_lpage_info *linfo; + + max_level =min (max_level, max_huge_page_level); +for ( ; max_level >PG_LEVEL_4K ; max_level--) { + linfo =lpage_info_slot (gfn, slot, max_level); +if (!linfo->disallow_lpage) +break ; + } + +if (max_level ==PG_LEVEL_4K ) +return PG_LEVEL_4K ; + +return host_pfn_mapping_level (kvm, gfn, pfn, slot); +} +int kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,gfn_t gfn,int max_level,kvm_pfn_t *pfnp,bool huge_page_disallowed,int *req_level) {struct kvm_memory_slot *slot; -struct kvm_lpage_info *linfo;kvm_pfn_t pfn = *pfnp;kvm_pfn_t mask;int level;
@@ -2805,17 +2822,7 @@ int kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, gfn_t gfn,if (!slot)return PG_LEVEL_4K ; - max_level =min (max_level, max_huge_page_level); -for ( ; max_level >PG_LEVEL_4K ; max_level--) { - linfo =lpage_info_slot (gfn, slot, max_level); -if (!linfo->disallow_lpage) -break ; - } - -if (max_level ==PG_LEVEL_4K ) -return PG_LEVEL_4K ; - - level =host_pfn_mapping_level (vcpu, gfn, pfn, slot); + level =kvm_mmu_max_mapping_level (vcpu->kvm, slot, gfn, pfn, max_level);if (level ==PG_LEVEL_4K )return level;
arch/x86/kvm/mmu/mmu_internal.h +2 -0
@@ -138,6 +138,8 @@ enum {#define SET_SPTE_NEED_REMOTE_TLB_FLUSH BIT(1)#define SET_SPTE_SPURIOUS BIT(2) +int kvm_mmu_max_mapping_level (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,kvm_pfn_t pfn,int max_level);int kvm_mmu_hugepage_adjust (struct kvm_vcpu *vcpu,gfn_t gfn,int max_level,kvm_pfn_t *pfnp,bool huge_page_disallowed,int *req_level);
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 4/14 KVM: x86/mmu: Pass the memslot to the rmap callbacks
In reply to Sean ChristophersonPass the memslot to the rmap callbacks, it will be used when zapping collapsible SPTEs to verify the memslot is compatible with hugepages before zapping its SPTEs.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-)
arch/x86/kvm/mmu/mmu.c +15 -9
@@ -1165,7 +1165,8 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep)* - W bit on ad-disabled SPTEs. * Returns true iff any D or W bits were cleared. */ -static bool __rmap_clear_dirty (struct kvm *kvm,struct kvm_rmap_head *rmap_head) +static bool __rmap_clear_dirty (struct kvm *kvm,struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot) {u64 *sptep;struct rmap_iterator iter;
@@ -1196,7 +1197,8 @@ static bool spte_set_dirty(u64 *sptep)return mmu_spte_update (sptep, spte); } -static bool __rmap_set_dirty (struct kvm *kvm,struct kvm_rmap_head *rmap_head) +static bool __rmap_set_dirty (struct kvm *kvm,struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot) {u64 *sptep;struct rmap_iterator iter;
@@ -1260,7 +1262,7 @@ void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm,while (mask) { rmap_head =__gfn_to_rmap (slot->base_gfn + gfn_offset +__ffs (mask),PG_LEVEL_4K , slot); -__rmap_clear_dirty (kvm, rmap_head); +__rmap_clear_dirty (kvm, rmap_head, slot);/* clear the first set bit */ mask &= mask -1 ;
@@ -1325,7 +1327,8 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn)return kvm_mmu_slot_gfn_write_protect (vcpu->kvm, slot, gfn); } -static bool kvm_zap_rmapp (struct kvm *kvm,struct kvm_rmap_head *rmap_head) +static bool kvm_zap_rmapp (struct kvm *kvm,struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot) {u64 *sptep;struct rmap_iterator iter;
@@ -1345,7 +1348,7 @@ static int kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, struct kvm_memory_slot *slot, gfn_t gfn, int level, unsigned long data) { - return kvm_zap_rmapp(kvm, rmap_head); + return kvm_zap_rmapp(kvm, rmap_head, slot); } static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
@@ -5189,7 +5192,8 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_max_root_level,EXPORT_SYMBOL_GPL (kvm_configure_mmu);/* The return value indicates if tlb flush on all vcpus is needed. */ -typedef bool (*slot_level_handler ) (struct kvm *kvm,struct kvm_rmap_head *rmap_head); +typedef bool (*slot_level_handler ) (struct kvm *kvm,struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot);/* The caller should hold mmu-lock before calling this function. */ static __always_inline bool
@@ -5203,7 +5207,7 @@ slot_handle_level_range(struct kvm *kvm, struct kvm_memory_slot *memslot,for_each_slot_rmap_range (memslot, start_level, end_level, start_gfn, end_gfn, &iterator) {if (iterator.rmap) - flush |=fn (kvm, iterator.rmap); + flush |=fn (kvm, iterator.rmap, memslot);if (need_resched () ||rwlock_needbreak (&kvm->mmu_lock)) {if (flush && lock_flush_tlb) {
@@ -5492,7 +5496,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) }static bool slot_rmap_write_protect (struct kvm *kvm, -struct kvm_rmap_head *rmap_head) +struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot) {return __rmap_write_protect (kvm, rmap_head, false); }
@@ -5526,7 +5531,8 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, }static bool kvm_mmu_zap_collapsible_spte (struct kvm *kvm, -struct kvm_rmap_head *rmap_head) +struct kvm_rmap_head *rmap_head, +struct kvm_memory_slot *slot) {u64 *sptep;struct rmap_iterator iter;
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 5/14 KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
In reply to Sean ChristophersonWhen zapping SPTEs in order to rebuild them as huge pages, use the new helper that computes the max mapping level to detect whether or not a SPTE should be zapped. Doing so avoids zapping SPTEs that can't possibly be rebuilt as huge pages, e.g. due to hardware constraints, memslot alignment, etc...
This also avoids zapping SPTEs that are still large, e.g. if migration was canceled before write-protected huge pages were shattered to enable dirty logging. Note, such pages are still write-protected at this time, i.e. a page fault VM-Exit will still occur. This will hopefully be addressed in a future patch.
Sadly, TDP MMU loses its const on the memslot, but that's a pervasive problem that's been around for quite some time.
--- arch/x86/kvm/mmu/mmu.c | 11 ++++++----- arch/x86/kvm/mmu/tdp_mmu.c | 13 +++++++------ arch/x86/kvm/mmu/tdp_mmu.h | 2 +- 3 files changed, 14 insertions(+), 12 deletions(-)
arch/x86/kvm/mmu/mmu.c +6 -5
@@ -5553,8 +5553,8 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,* mapping if the indirect sp has level = 1. */ if (sp->role.direct && !kvm_is_reserved_pfn (pfn) && - (kvm_is_zone_device_pfn (pfn) || -PageCompound (pfn_to_page (pfn)))) { + sp->role.level <kvm_mmu_max_mapping_level (kvm, slot, sp->gfn, + pfn,PG_LEVEL_NUM )) {pte_list_remove (rmap_head, sptep);if (kvm_available_flush_tlb_with_range ())
@@ -5574,12 +5574,13 @@ void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot) {/* FIXME: const-ify all uses of struct kvm_memory_slot. */ +struct kvm_memory_slot *slot = (struct kvm_memory_slot *)memslot; +write_lock (&kvm->mmu_lock); -slot_handle_leaf (kvm, (struct kvm_memory_slot *)memslot, - kvm_mmu_zap_collapsible_spte, true); +slot_handle_leaf (kvm, slot, kvm_mmu_zap_collapsible_spte, true);if (is_tdp_mmu_enabled (kvm)) -kvm_tdp_mmu_zap_collapsible_sptes (kvm, memslot); +kvm_tdp_mmu_zap_collapsible_sptes (kvm, slot);write_unlock (&kvm->mmu_lock); }
arch/x86/kvm/mmu/tdp_mmu.c +7 -6
@@ -1328,8 +1328,10 @@ bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot)*/ static void zap_collapsible_spte_range (struct kvm *kvm,struct kvm_mmu_page *root, -gfn_t start,gfn_t end) +struct kvm_memory_slot *slot) { +gfn_t start = slot->base_gfn; +gfn_t end = start + slot->npages;struct tdp_iter iter;kvm_pfn_t pfn;bool spte_set = false;
@@ -1348,8 +1350,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - (!PageTransCompoundMap (pfn_to_page (pfn)) && - !kvm_is_zone_device_pfn (pfn))) + iter.level >=kvm_mmu_max_mapping_level (kvm, slot, iter.gfn, + pfn,PG_LEVEL_NUM ))continue ;tdp_mmu_set_spte (kvm, &iter,0 );
@@ -1367,7 +1369,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm,* be replaced by large mappings, for GFNs within the slot. */ void kvm_tdp_mmu_zap_collapsible_sptes (struct kvm *kvm, -const struct kvm_memory_slot *slot) +struct kvm_memory_slot *slot) {struct kvm_mmu_page *root;int root_as_id;
@@ -1377,8 +1379,7 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm,if (root_as_id != slot->as_id)continue ; -zap_collapsible_spte_range (kvm, root, slot->base_gfn, - slot->base_gfn + slot->npages); +zap_collapsible_spte_range (kvm, root, slot); } }
arch/x86/kvm/mmu/tdp_mmu.h +1 -1
@@ -35,7 +35,7 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm, bool wrprot);bool kvm_tdp_mmu_slot_set_dirty (struct kvm *kvm,struct kvm_memory_slot *slot);void kvm_tdp_mmu_zap_collapsible_sptes (struct kvm *kvm, -const struct kvm_memory_slot *slot); +struct kvm_memory_slot *slot);bool kvm_tdp_mmu_write_protect_gfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn);
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Paolo Bonzini
maintainerRe: [PATCH 05/14] KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
arch/x86/kvm/mmu/tdp_mmu.c static void zap_collapsible_spte_range(struct kvm *kvm, · line 1355
Quoted 9 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +2
-2pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - (!PageTransCompoundMap (pfn_to_page (pfn)) && - !kvm_is_zone_device_pfn (pfn))) + iter.level >=kvm_mmu_max_mapping_level (kvm, slot, iter.gfn, + pfn,PG_LEVEL_NUM ))continue ;
This changes the test to PageCompound. Is it worth moving the change to patch 1?
Paolo
Sean Christopherson
maintainerRe: [PATCH 05/14] KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
Quoted 13 lines · Paolo Bonzini
arch/x86/kvm/mmu/tdp_mmu.c static void zap_collapsible_spte_range(struct kvm *kvm, · line 1355
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +2
-2pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - (!PageTransCompoundMap (pfn_to_page (pfn)) && - !kvm_is_zone_device_pfn (pfn))) + iter.level >=kvm_mmu_max_mapping_level (kvm, slot, iter.gfn, + pfn,PG_LEVEL_NUM ))continue ;This changes the test to PageCompound. Is it worth moving the change to patch 1?
Yes? I originally did that in a separate patch, then changed my mind.
If PageTransCompoundMap() also detects HugeTLB pages, then it is the "better" option as it checks that the page is actually mapped huge. I dropped the change because PageTransCompound() is just a wrapper around PageCompound(), and so I assumed PageTransCompoundMap() would detect HugeTLB pages, too. I'm not so sure about that after rereading the code, yet again.
Re: [PATCH 05/14] KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
Quoted 15 lines · Sean Christopherson
Quoted 13 lines · Paolo Bonzini
arch/x86/kvm/mmu/tdp_mmu.c static void zap_collapsible_spte_range(struct kvm *kvm, · line 1355
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +2
-2pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - (!PageTransCompoundMap (pfn_to_page (pfn)) && - !kvm_is_zone_device_pfn (pfn))) + iter.level >=kvm_mmu_max_mapping_level (kvm, slot, iter.gfn, + pfn,PG_LEVEL_NUM ))continue ;This changes the test to PageCompound. Is it worth moving the change to patch 1?
Quoted 7 lines
Yes? I originally did that in a separate patch, then changed my mind.
If
PageTransCompoundMap()also detects HugeTLB pages, then it is the "better" option as it checks that the page is actually mapped huge. I dropped the change becausePageTransCompound()is just a wrapper aroundPageCompound(), and so I assumedPageTransCompoundMap()would detect HugeTLB pages, too. I'm not so sure about that after rereading the code, yet again.
I have not followed this thread, but HugeTLB hit my mail filter and I can help with this question.
No, PageTransCompoundMap() will not detect HugeTLB. hugetlb pages do not use the compound_mapcount_ptr field. So, that final check/return in PageTransCompoundMap() will always be false.
Signature · 1 line
Mike Kravetz
Sean Christopherson
maintainerRe: [PATCH 05/14] KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
Quoted 24 lines · Mike Kravetz
Quoted 15 lines · Sean Christopherson
Quoted 13 lines · Paolo Bonzini
arch/x86/kvm/mmu/tdp_mmu.c static void zap_collapsible_spte_range(struct kvm *kvm, · line 1355
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/tdp_mmu.c +2
-2pfn =spte_to_pfn (iter.old_spte);if (kvm_is_reserved_pfn (pfn) || - (!PageTransCompoundMap (pfn_to_page (pfn)) && - !kvm_is_zone_device_pfn (pfn))) + iter.level >=kvm_mmu_max_mapping_level (kvm, slot, iter.gfn, + pfn,PG_LEVEL_NUM ))continue ;This changes the test to PageCompound. Is it worth moving the change to patch 1?
Quoted 7 lines
Yes? I originally did that in a separate patch, then changed my mind.
If
PageTransCompoundMap()also detects HugeTLB pages, then it is the "better" option as it checks that the page is actually mapped huge. I dropped the change becausePageTransCompound()is just a wrapper aroundPageCompound(), and so I assumedPageTransCompoundMap()would detect HugeTLB pages, too. I'm not so sure about that after rereading the code, yet again.
Quoted 6 lines
I have not followed this thread, but HugeTLB hit my mail filter and I can help with this question.
No,
PageTransCompoundMap()will not detect HugeTLB. hugetlb pages do not use thecompound_mapcount_ptrfield. So, that final check/return inPageTransCompoundMap()will always be false.
Thanks Mike!
Paolo, I agree it makes sense to switch to PageCompound in the earlier patch, in case this one needs to be reverted.
Sean Christopherson
maintainerPATCH 6/14 KVM: nVMX: Disable PML in hardware when running L2
In reply to Sean ChristophersonUnconditionally disable PML in vmcs02, KVM emulates PML purely in the MMU, e.g. vmx_flush_pml_buffer() doesn't even try to copy the L2 GPAs from vmcs02's buffer to vmcs12. At best, enabling PML is a nop. At worst, it will cause vmx_flush_pml_buffer() to record bogus GFNs in the dirty logs.
Initialize vmcs02.GUEST_PML_INDEX such that PML writes would trigger VM-Exit if PML was somehow enabled, skip flushing the buffer for guest mode since the index is bogus, and freak out if a PML full exit occurs when L2 is active.
--- arch/x86/kvm/vmx/nested.c | 29 +++++++++++++++-------------- arch/x86/kvm/vmx/vmx.c | 12 ++++++++++-- 2 files changed, 25 insertions(+), 16 deletions(-)
arch/x86/kvm/vmx/nested.c +15 -14
@@ -2167,15 +2167,13 @@ static void prepare_vmcs02_constant_state(struct vcpu_vmx *vmx)vmcs_write64 (MSR_BITMAP ,__pa (vmx->nested.vmcs02.msr_bitmap));/* -* The PML address never changes, so it is constant in vmcs02. -* Conceptually we want to copy the PML index from vmcs01 here, -* and then back to vmcs01 on nested vmexit. But since we flush -* the log and reset GUEST_PML_INDEX on each vmexit, the PML -* index is also effectively constant in vmcs02. +* PML is emulated for L2, but never enabled in hardware as the MMU +* handles A/D emulation. Disabling PML for L2 also avoids having to +* deal with filtering out L2 GPAs from the buffer. */ if (enable_pml) { -vmcs_write64 (PML_ADDRESS ,page_to_phys (vmx->pml_pg)); -vmcs_write16 (GUEST_PML_INDEX ,PML_ENTITY_NUM -1 ); +vmcs_write64 (PML_ADDRESS ,0 ); +vmcs_write16 (GUEST_PML_INDEX ,-1 ); }if (cpu_has_vmx_encls_vmexit ())
@@ -2210,7 +2208,7 @@ static void prepare_vmcs02_early_rare(struct vcpu_vmx *vmx,static void prepare_vmcs02_early (struct vcpu_vmx *vmx,struct vmcs12 *vmcs12) { -u32 exec_control, vmcs12_exec_ctrl; +u32 exec_control;u64 guest_efer =nested_vmx_calc_efer (vmx, vmcs12);if (vmx->nested.dirty_vmcs12 || vmx->nested.hv_evmcs)
@@ -2284,11 +2282,11 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct vmcs12 *vmcs12)SECONDARY_EXEC_APIC_REGISTER_VIRT | SECONDARY_EXEC_ENABLE_VMFUNC);if (nested_cpu_has (vmcs12, -CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )) { - vmcs12_exec_ctrl = vmcs12->secondary_vm_exec_control & - ~SECONDARY_EXEC_ENABLE_PML ; - exec_control |= vmcs12_exec_ctrl; - } +CPU_BASED_ACTIVATE_SECONDARY_CONTROLS )) + exec_control |= vmcs12->secondary_vm_exec_control; + +/* PML is emulated and never enabled in hardware for L2. */ + exec_control &= ~SECONDARY_EXEC_ENABLE_PML ;/* VMCS shadowing for L2 is emulated for now */ exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS ;
@@ -5793,7 +5791,10 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu *vcpu,case EXIT_REASON_PREEMPTION_TIMER :return true;case EXIT_REASON_PML_FULL : -/* We emulate PML support to L1. */ +/* +* PML is emulated for an L1 VMM and should never be enabled in +* vmcs02, always "handle" PML_FULL by exiting to userspace. +*/ return true;case EXIT_REASON_VMFUNC :/* VM functions are emulated through L2->L0 vmexits. */
arch/x86/kvm/vmx/vmx.c +10 -2
@@ -5976,9 +5976,10 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)* updated. Another good is, in kvm_vm_ioctl_get_dirty_log, before * querying dirty_bitmap, we only need to kick all vcpus out of guest * mode as if vcpus is in root mode, the PML buffer must has been -* flushed already. +* flushed already. Note, PML is never enabled in hardware while +* running L2. */ -if (enable_pml) +if (enable_pml && !is_guest_mode (vcpu))vmx_flush_pml_buffer (vcpu);/*
@@ -5994,6 +5995,13 @@ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)return handle_invalid_guest_state (vcpu);if (is_guest_mode (vcpu)) { +/* +* PML is never enabled when running L2, bail immediately if a +* PML full exit occurs as something is horribly wrong. +*/ +if (exit_reason.basic ==EXIT_REASON_PML_FULL ) + goto unexpected_vmexit; +/* * The host physical addresses of some pages of guest memory * are loaded into the vmcs02 (e.g. vmcs12's Virtual APIC
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 7/14 KVM: x86/mmu: Expand on the comment in kvm_vcpu_ad_need_write_protect()
In reply to Sean ChristophersonExpand the comment about need to use write-protection for nested EPT when PML is enabled to clarify that the tagging is a nop when PML is _not_ enabled. Without the clarification, omitting the PML check looks wrong at first^Wfifth glance.
--- arch/x86/kvm/mmu/mmu_internal.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/mmu_internal.h +4 -1
@@ -84,7 +84,10 @@ static inline bool kvm_vcpu_ad_need_write_protect(struct kvm_vcpu *vcpu)* When using the EPT page-modification log, the GPAs in the log * would come from L2 rather than L1. Therefore, we need to rely * on write protection to record dirty pages. This also bypasses -* PML, since writes now result in a vmexit. +* PML, since writes now result in a vmexit. Note, this helper will +* tag SPTEs as needing write-protection even if PML is disabled or +* unsupported, but that's ok because the tag is consumed if and only +* if PML is enabled. Omit the PML check to save a few uops. */ return vcpu->arch.mmu == &vcpu->arch.guest_mmu; }
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 8/14 KVM: x86/mmu: Make dirty log size hook (PML) a value, not a function
In reply to Sean ChristophersonStore the vendor-specific dirty log size in a variable, there's no need to wrap it in a function since the value is constant after hardware_setup() runs.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 - arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 5 +---- arch/x86/kvm/vmx/vmx.c | 9 ++------- 4 files changed, 4 insertions(+), 13 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +0 -1
@@ -97,7 +97,6 @@ KVM_X86_OP_NULL(slot_enable_log_dirty) KVM_X86_OP_NULL(slot_disable_log_dirty) KVM_X86_OP_NULL(flush_log_dirty) KVM_X86_OP_NULL(enable_log_dirty_pt_masked) -KVM_X86_OP_NULL(cpu_dirty_log_size) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1294,7 +1294,7 @@ struct kvm_x86_ops {void (*enable_log_dirty_pt_masked)(struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t offset,unsigned long mask); -int (*cpu_dirty_log_size)(void ); +int cpu_dirty_log_size;/* pmu operations of sub-arch */ const struct kvm_pmu_ops *pmu_ops;
arch/x86/kvm/mmu/mmu.c +1 -4
@@ -1294,10 +1294,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,int kvm_cpu_dirty_log_size (void ) { -if (kvm_x86_ops.cpu_dirty_log_size) -return static_call (kvm_x86_cpu_dirty_log_size)(); - -return 0 ; +return kvm_x86_ops.cpu_dirty_log_size; } bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,
arch/x86/kvm/vmx/vmx.c +2 -7
@@ -7650,11 +7650,6 @@ static bool vmx_check_apicv_inhibit_reasons(ulong bit)return supported &BIT (bit); } -static int vmx_cpu_dirty_log_size (void ) -{ -return enable_pml ?PML_ENTITY_NUM :0 ; -} -static struct kvm_x86_ops vmx_x86_ops__initdata = { .hardware_unsetup = hardware_unsetup,
@@ -7758,6 +7753,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .slot_disable_log_dirty = vmx_slot_disable_log_dirty, .flush_log_dirty = vmx_flush_log_dirty, .enable_log_dirty_pt_masked = vmx_enable_log_dirty_pt_masked, + .cpu_dirty_log_size = PML_ENTITY_NUM, .pre_block = vmx_pre_block, .post_block = vmx_post_block,
@@ -7785,7 +7781,6 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .msr_filter_changed = vmx_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .cpu_dirty_log_size = vmx_cpu_dirty_log_size, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, };
@@ -7907,7 +7902,7 @@ static __init int hardware_setup(void) vmx_x86_ops.slot_disable_log_dirty =NULL ; vmx_x86_ops.flush_log_dirty =NULL ; vmx_x86_ops.enable_log_dirty_pt_masked =NULL ; - vmx_x86_ops.cpu_dirty_log_size =NULL ; + vmx_x86_ops.cpu_dirty_log_size =0 ; }if (!cpu_has_vmx_preemption_timer ())
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Paolo Bonzini
maintainerRe: [PATCH 08/14] KVM: x86/mmu: Make dirty log size hook (PML) a value, not a function
Quoted 3 lines · Sean Christopherson
Store the vendor-specific dirty log size in a variable, there's no need to wrap it in a function since the value is constant after
hardware_setup()runs.
For now... :)
Paolo
Quoted 91 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm-x86-ops.h | 1 - arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/mmu/mmu.c | 5 +---- arch/x86/kvm/vmx/vmx.c | 9 ++------- 4 files changed, 4 insertions(+), 13 deletions(-)arch/x86/include/asm/kvm-x86-ops.h +0
-1@@ -97,7 +97,6 @@ KVM_X86_OP_NULL(slot_enable_log_dirty) KVM_X86_OP_NULL(slot_disable_log_dirty) KVM_X86_OP_NULL(flush_log_dirty) KVM_X86_OP_NULL(enable_log_dirty_pt_masked) -KVM_X86_OP_NULL(cpu_dirty_log_size) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking)arch/x86/include/asm/kvm_host.h +1
-1@@ -1294,7 +1294,7 @@ struct kvm_x86_ops {void (*enable_log_dirty_pt_masked)(struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t offset,unsigned long mask); -int (*cpu_dirty_log_size)(void ); +int cpu_dirty_log_size;/* pmu operations of sub-arch */ const struct kvm_pmu_ops *pmu_ops;arch/x86/kvm/mmu/mmu.c +1
-4@@ -1294,10 +1294,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,int kvm_cpu_dirty_log_size (void ) { -if (kvm_x86_ops.cpu_dirty_log_size) -return static_call (kvm_x86_cpu_dirty_log_size)(); - -return 0 ; +return kvm_x86_ops.cpu_dirty_log_size; } bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm,arch/x86/kvm/vmx/vmx.c +2
-7@@ -7650,11 +7650,6 @@ static bool vmx_check_apicv_inhibit_reasons(ulong bit)return supported &BIT (bit); } -static int vmx_cpu_dirty_log_size (void ) -{ -return enable_pml ?PML_ENTITY_NUM :0 ; -} -static struct kvm_x86_ops vmx_x86_ops__initdata = { .hardware_unsetup = hardware_unsetup,@@ -7758,6 +7753,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .slot_disable_log_dirty = vmx_slot_disable_log_dirty, .flush_log_dirty = vmx_flush_log_dirty, .enable_log_dirty_pt_masked = vmx_enable_log_dirty_pt_masked, + .cpu_dirty_log_size = PML_ENTITY_NUM, .pre_block = vmx_pre_block, .post_block = vmx_post_block,@@ -7785,7 +7781,6 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .msr_filter_changed = vmx_msr_filter_changed, .complete_emulated_msr = kvm_complete_insn_gp, - .cpu_dirty_log_size = vmx_cpu_dirty_log_size, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, };@@ -7907,7 +7902,7 @@ static __init int hardware_setup(void) vmx_x86_ops.slot_disable_log_dirty =NULL ; vmx_x86_ops.flush_log_dirty =NULL ; vmx_x86_ops.enable_log_dirty_pt_masked =NULL ; - vmx_x86_ops.cpu_dirty_log_size =NULL ; + vmx_x86_ops.cpu_dirty_log_size =0 ; }if (!cpu_has_vmx_preemption_timer ())
Sean Christopherson
maintainerPATCH 9/14 KVM: x86: Move MMU's PML logic to common code
In reply to Sean ChristophersonDrop the facade of KVM's PML logic being vendor specific and move the bits that aren't truly VMX specific into common x86 code. The MMU logic for dealing with PML is tightly coupled to the feature and to VMX's implementation, bouncing through kvm_x86_ops obfuscates the code without providing any meaningful separation of concerns or encapsulation.
No functional change intended.
--- arch/x86/include/asm/kvm-x86-ops.h | 4 --- arch/x86/include/asm/kvm_host.h | 27 ++------------- arch/x86/kvm/mmu/mmu.c | 16 +++------ arch/x86/kvm/vmx/vmx.c | 55 +----------------------------- arch/x86/kvm/x86.c | 22 ++++++++---- 5 files changed, 24 insertions(+), 100 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +0 -4
@@ -93,10 +93,6 @@ KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff) KVM_X86_OP_NULL(request_immediate_exit) KVM_X86_OP(sched_in) -KVM_X86_OP_NULL(slot_enable_log_dirty) -KVM_X86_OP_NULL(slot_disable_log_dirty) -KVM_X86_OP_NULL(flush_log_dirty) -KVM_X86_OP_NULL(enable_log_dirty_pt_masked) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking)
arch/x86/include/asm/kvm_host.h +2 -25
@@ -1271,29 +1271,9 @@ struct kvm_x86_ops {void (*sched_in)(struct kvm_vcpu *kvm,int cpu);/* -* Arch-specific dirty logging hooks. These hooks are only supposed to -* be valid if the specific arch has hardware-accelerated dirty logging -* mechanism. Currently only for PML on VMX. -* -* - slot_enable_log_dirty: -* called when enabling log dirty mode for the slot. -* - slot_disable_log_dirty: -* called when disabling log dirty mode for the slot. -* also called when slot is created with log dirty disabled. -* - flush_log_dirty: -* called before reporting dirty_bitmap to userspace. -* - enable_log_dirty_pt_masked: -* called when reenabling log dirty for the GFNs in the mask after -* corresponding bits are cleared in slot->dirty_bitmap. +* Size of the CPU's dirty log buffer, i.e. VMX's PML buffer. A zero +* value indicates CPU dirty logging is unsupported or disabled. */ -void (*slot_enable_log_dirty)(struct kvm *kvm, -struct kvm_memory_slot *slot); -void (*slot_disable_log_dirty)(struct kvm *kvm, -struct kvm_memory_slot *slot); -void (*flush_log_dirty)(struct kvm *kvm); -void (*enable_log_dirty_pt_masked)(struct kvm *kvm, -struct kvm_memory_slot *slot, -gfn_t offset,unsigned long mask);int cpu_dirty_log_size;/* pmu operations of sub-arch */
@@ -1439,9 +1419,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm, struct kvm_memory_slot *memslot);void kvm_mmu_slot_set_dirty (struct kvm *kvm,struct kvm_memory_slot *memslot); -void kvm_mmu_clear_dirty_pt_masked (struct kvm *kvm, -struct kvm_memory_slot *slot, -gfn_t gfn_offset,unsigned long mask);void kvm_mmu_zap_all (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen);unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm);
arch/x86/kvm/mmu/mmu.c +5 -11
@@ -1250,9 +1250,9 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,* * Used for PML to re-log the dirty GPAs after userspace querying dirty_bitmap. */ -void kvm_mmu_clear_dirty_pt_masked (struct kvm *kvm, -struct kvm_memory_slot *slot, -gfn_t gfn_offset,unsigned long mask) +static void kvm_mmu_clear_dirty_pt_masked (struct kvm *kvm, +struct kvm_memory_slot *slot, +gfn_t gfn_offset,unsigned long mask) {struct kvm_rmap_head *rmap_head;
@@ -1268,7 +1268,6 @@ void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, mask &= mask -1 ; } } -EXPORT_SYMBOL_GPL (kvm_mmu_clear_dirty_pt_masked);/** * kvm_arch_mmu_enable_log_dirty_pt_masked - enable dirty logging for selected
@@ -1284,10 +1283,8 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { - if (kvm_x86_ops.enable_log_dirty_pt_masked) -static_call (kvm_x86_enable_log_dirty_pt_masked)(kvm ,slot , -gfn_offset , -mask ); + if (kvm_x86_ops.cpu_dirty_log_size) + kvm_mmu_clear_dirty_pt_masked(kvm, slot, gfn_offset, mask); elsekvm_mmu_write_protect_pt_masked (kvm ,slot ,gfn_offset ,mask ); }
@@ -5616,7 +5613,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,if (flush)kvm_arch_flush_remote_tlbs_memslot (kvm, memslot); } -EXPORT_SYMBOL_GPL (kvm_mmu_slot_leaf_clear_dirty);void kvm_mmu_slot_largepage_remove_write_access (struct kvm *kvm,struct kvm_memory_slot *memslot)
@@ -5633,7 +5629,6 @@ void kvm_mmu_slot_largepage_remove_write_access(struct kvm *kvm,if (flush)kvm_arch_flush_remote_tlbs_memslot (kvm, memslot); } -EXPORT_SYMBOL_GPL (kvm_mmu_slot_largepage_remove_write_access);void kvm_mmu_slot_set_dirty (struct kvm *kvm,struct kvm_memory_slot *memslot)
@@ -5649,7 +5644,6 @@ void kvm_mmu_slot_set_dirty(struct kvm *kvm,if (flush)kvm_arch_flush_remote_tlbs_memslot (kvm, memslot); } -EXPORT_SYMBOL_GPL (kvm_mmu_slot_set_dirty);void kvm_mmu_zap_all (struct kvm *kvm) {
arch/x86/kvm/vmx/vmx.c +1 -54
80 lines not on this page · raw message
arch/x86/kvm/x86.c +16 -6
@@ -5214,10 +5214,18 @@ static int kvm_vm_ioctl_reinject(struct kvm *kvm,void kvm_arch_sync_dirty_log (struct kvm *kvm,struct kvm_memory_slot *memslot) { +/* -* Flush potentially hardware-cached dirty pages to dirty_bitmap. +* Flush all CPUs' dirty log buffers to the dirty_bitmap. Called +* before reporting dirty_bitmap to userspace. KVM flushes the buffers +* on all VM-Exits, thus we only need to kick running vCPUs to force a +* VM-Exit. */ -static_call_cond (kvm_x86_flush_log_dirty)(kvm); +struct kvm_vcpu *vcpu; +int i; + +kvm_for_each_vcpu (i, vcpu, kvm) +kvm_vcpu_kick (vcpu); } int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_event,
@@ -10809,8 +10817,10 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* is enabled the D-bit or the W-bit will be cleared. */ if (new->flags &KVM_MEM_LOG_DIRTY_PAGES ) { -if (kvm_x86_ops.slot_enable_log_dirty) { -static_call (kvm_x86_slot_enable_log_dirty)(kvm, new); +if (kvm_x86_ops.cpu_dirty_log_size) { +if (!kvm_dirty_log_manual_protect_and_init_set (kvm)) +kvm_mmu_slot_leaf_clear_dirty (kvm, new); +kvm_mmu_slot_largepage_remove_write_access (kvm, new); }else {int level = kvm_dirty_log_manual_protect_and_init_set(kvm) ?
@@ -10826,8 +10836,8 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,*/ kvm_mmu_slot_remove_write_access (kvm, new, level); } - }else { -static_call_cond (kvm_x86_slot_disable_log_dirty)(kvm, new); + }else if (kvm_x86_ops.cpu_dirty_log_size) { +kvm_mmu_slot_set_dirty (kvm, new); } }
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 10/14 KVM: x86: Further clarify the logic and comments for toggling log dirty
In reply to Sean ChristophersonAdd a sanity check in kvm_mmu_slot_apply_flags to assert that the LOG_DIRTY_PAGES flag is indeed being toggled, and explicitly rely on that holding true when zapping collapsible SPTEs. Manipulating the CPU dirty log (PML) and write-protection also relies on this assertion, but that's not obvious in the current code.
--- arch/x86/kvm/x86.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
arch/x86/kvm/x86.c +11 -4
@@ -10761,12 +10761,20 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, enum kvm_mr_change change) {/* -* Nothing to do for RO slots or CREATE/MOVE/DELETE of a slot. -* See comments below. +* Nothing to do for RO slots (which can't be dirtied and can't be made +* writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY ))return ; +/* +* READONLY and non-flags changes were filtered out above, and the only +* other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty +* logging isn't being toggled on or off. +*/ +if (WARN_ON_ONCE (!((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ))) +return ; +/* * Dirty logging tracks sptes in 4k granularity, meaning that large * sptes have to be split. If live migration is successful, the guest
@@ -10784,8 +10792,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* MOVE/DELETE: The old mappings will already have been cleaned up by * kvm_arch_flush_shadow_memslot() */ -if ((old->flags &KVM_MEM_LOG_DIRTY_PAGES ) && - !(new->flags &KVM_MEM_LOG_DIRTY_PAGES )) +if (!(new->flags &KVM_MEM_LOG_DIRTY_PAGES ))kvm_mmu_zap_collapsible_sptes (kvm, new);/*
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Paolo Bonzini
maintainerRe: [PATCH 10/14] KVM: x86: Further clarify the logic and comments for toggling log dirty
Quoted 7 lines · Sean Christopherson
arch/x86/kvm/x86.c +2
-2-* Nothing to do for RO slots or CREATE/MOVE/DELETE of a slot. -* See comments below. +* Nothing to do for RO slots (which can't be dirtied and can't be made +* writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY ))
arch/x86/kvm/x86.c static void kvm_mmu_slot_apply_flags(struct kvm *kvm, · line 10777
Quoted 10 lines
arch/x86/kvm/x86.c +8
-0return ; +/* +* READONLY and non-flags changes were filtered out above, and the only +* other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty +* logging isn't being toggled on or off. +*/ +if (WARN_ON_ONCE (!((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ))) +return ; +
What about readonly -> readwrite changes?
Paolo
Sean Christopherson
maintainerRe: [PATCH 10/14] KVM: x86: Further clarify the logic and comments for toggling log dirty
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Sean Christopherson
arch/x86/kvm/x86.c +4
-2-* Nothing to do for RO slots or CREATE/MOVE/DELETE of a slot. -* See comments below. +* Nothing to do for RO slots (which can't be dirtied and can't be made +* writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY ))return ; +/* +* READONLY and non-flags changes were filtered out above, and the only
Quoted 8 lines
arch/x86/kvm/x86.c static void kvm_mmu_slot_apply_flags(struct kvm *kvm, · line 10777
Quoted 6 lines
arch/x86/kvm/x86.c +6
-0+* other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty +* logging isn't being toggled on or off. +*/ +if (WARN_ON_ONCE (!((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ))) +return ; +What about readonly -> readwrite changes?
Not allowed without first deleting the memslot. See commit 75d61fbcf563 ("KVM:set_memory_region: Disallow changing read-only attribute later"). RW->RO is also not supported.
if (!old.npages) {
change = KVM_MR_CREATE;
new.dirty_bitmap = NULL;
memset(&new.arch, 0, sizeof(new.arch));
} else { /* Modify an existing slot. */
if ((new.userspace_addr != old.userspace_addr) ||
(new.npages != old.npages) ||
((new.flags ^ old.flags) & KVM_MEM_READONLY))
return -EINVAL;Paolo Bonzini
maintainerRe: [PATCH 10/14] KVM: x86: Further clarify the logic and comments for toggling log dirty
Quoted 25 lines · Sean Christopherson
Quoted 11 lines · Paolo Bonzini
Quoted 10 lines · Sean Christopherson
arch/x86/kvm/x86.c +4
-2-* Nothing to do for RO slots or CREATE/MOVE/DELETE of a slot. -* See comments below. +* Nothing to do for RO slots (which can't be dirtied and can't be made +* writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY ))return ; +/* +* READONLY and non-flags changes were filtered out above, and the only Quoted 8 lines
arch/x86/kvm/x86.c static void kvm_mmu_slot_apply_flags(struct kvm *kvm, · line 10777
Quoted 6 lines
arch/x86/kvm/x86.c +6
-0+* other flag is LOG_DIRTY_PAGES, i.e. something is wrong if dirty +* logging isn't being toggled on or off. +*/ +if (WARN_ON_ONCE (!((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ))) +return ; +What about readonly -> readwrite changes?
Not allowed without first deleting the memslot. See commit
75d61fbcf563("KVM:set_memory_region: Disallow changing read-only attribute later").RW->ROis also not supported.
Quoted 10 lines
if (!old.npages) {
change = KVM_MR_CREATE;
new.dirty_bitmap = NULL;
memset(&new.arch, 0, sizeof(new.arch));
} else { /* Modify an existing slot. */
if ((new.userspace_addr != old.userspace_addr) ||
(new.npages != old.npages) ||
((new.flags ^ old.flags) & KVM_MEM_READONLY))
return -EINVAL;Right you are, thanks. Then the warning would catch this quick. I queued 10 and 11 too, and will reply on 12 now that I looked at it more closely.
Paolo
Sean Christopherson
maintainerPATCH 11/14 KVM: VMX: Dynamically enable/disable PML based on memslot dirty logging
In reply to Sean ChristophersonFrom: Makarand Sonare <[email protected]>
Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Setting the dirty bits is an expensive operation which can cause severe MMU lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration.
Manually setting dirty bits also fails to prevent PML activity if some code path clears dirty bits, which can incur unnecessary VM-Exits.
In order to avoid this extra overhead, dynamically enable/disable PML when dirty logging gets turned on/off for the first/last memslot.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/vmx/nested.c | 5 +++++ arch/x86/kvm/vmx/vmx.c | 28 +++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 2 ++ arch/x86/kvm/x86.c | 35 ++++++++++++++++++++++++++---- 6 files changed, 70 insertions(+), 5 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -93,6 +93,7 @@ KVM_X86_OP(check_intercept) KVM_X86_OP(handle_exit_irqoff) KVM_X86_OP_NULL(request_immediate_exit) KVM_X86_OP(sched_in) +KVM_X86_OP_NULL(update_cpu_dirty_logging) KVM_X86_OP_NULL(pre_block) KVM_X86_OP_NULL(post_block) KVM_X86_OP_NULL(vcpu_blocking)
arch/x86/include/asm/kvm_host.h +4 -0
@@ -89,6 +89,8 @@ KVM_ARCH_REQ_FLAGS(27, KVM_REQUEST_NO_WAKEUP)#define KVM_REQ_APF_READY KVM_ARCH_REQ(28)#define KVM_REQ_MSR_FILTER_CHANGED KVM_ARCH_REQ(29) +#define KVM_REQ_UPDATE_CPU_DIRTY_LOGGING \ + KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)#define CR0_RESERVED_BITS \ (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
@@ -1007,6 +1009,7 @@ struct kvm_arch {u32 bsp_vcpu_id;u64 disabled_quirks; +int cpu_dirty_logging_count;enum kvm_irqchip_mode irqchip_mode;u8 nr_reserved_ioapic_pins;
@@ -1275,6 +1278,7 @@ struct kvm_x86_ops {* value indicates CPU dirty logging is unsupported or disabled. */ int cpu_dirty_log_size; +void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu);/* pmu operations of sub-arch */ const struct kvm_pmu_ops *pmu_ops;
arch/x86/kvm/vmx/nested.c +5 -0
@@ -4493,6 +4493,11 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,vmx_set_virtual_apic_mode (vcpu); } +if (vmx->nested.update_vmcs01_cpu_dirty_logging) { + vmx->nested.update_vmcs01_cpu_dirty_logging = false; +vmx_update_cpu_dirty_logging (vcpu); + } +/* Unpin physical memory we referred to in vmcs02 */ if (vmx->nested.apic_access_page) {kvm_release_page_clean (vmx->nested.apic_access_page);
arch/x86/kvm/vmx/vmx.c +27 -1
@@ -4277,7 +4277,12 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)*/ exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS ; -if (!enable_pml) +/* +* PML is enabled/disabled when dirty logging of memsmlots changes, but +* it needs to be set here when dirty logging is already active, e.g. +* if this vCPU was created after dirty logging was enabled. +*/ +if (!vcpu->kvm->arch.cpu_dirty_logging_count) exec_control &= ~SECONDARY_EXEC_ENABLE_PML ;if (cpu_has_vmx_xsaves ()) {
@@ -7499,6 +7504,26 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu)shrink_ple_window (vcpu); } +void vmx_update_cpu_dirty_logging (struct kvm_vcpu *vcpu) +{ +struct vcpu_vmx *vmx =to_vmx (vcpu); + +if (is_guest_mode (vcpu)) { + vmx->nested.update_vmcs01_cpu_dirty_logging = true; +return ; + } + +/* +* Note, cpu_dirty_logging_count can be changed concurrent with this +* code, but in that case another update request will be made and so +* the guest will never run with a stale PML value. +*/ +if (vcpu->kvm->arch.cpu_dirty_logging_count) +secondary_exec_controls_setbit (vmx,SECONDARY_EXEC_ENABLE_PML ); +else +secondary_exec_controls_clearbit (vmx,SECONDARY_EXEC_ENABLE_PML ); +} +static int vmx_pre_block (struct kvm_vcpu *vcpu) {if (pi_pre_block (vcpu))
@@ -7706,6 +7731,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .sched_in = vmx_sched_in, .cpu_dirty_log_size = PML_ENTITY_NUM, + .update_cpu_dirty_logging = vmx_update_cpu_dirty_logging, .pre_block = vmx_pre_block, .post_block = vmx_post_block,
arch/x86/kvm/vmx/vmx.h +2 -0
@@ -165,6 +165,7 @@ struct nested_vmx {bool change_vmcs01_virtual_apic_mode;bool reload_vmcs01_apic_access_page; +bool update_vmcs01_cpu_dirty_logging;/* * Enlightened VMCS has been enabled. It does not mean that L1 has to
@@ -393,6 +394,7 @@ int vmx_find_loadstore_msr_slot(struct vmx_msrs *m, u32 msr);void vmx_ept_load_pdptrs (struct kvm_vcpu *vcpu);void vmx_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type,bool value); +void vmx_update_cpu_dirty_logging (struct kvm_vcpu *vcpu);static inline u8 vmx_get_rvi (void ) {
arch/x86/kvm/x86.c +31 -4
@@ -8987,6 +8987,9 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)kvm_check_async_pf_completion (vcpu);if (kvm_check_request (KVM_REQ_MSR_FILTER_CHANGED , vcpu))static_call (kvm_x86_msr_filter_changed)(vcpu); + +if (kvm_check_request (KVM_REQ_UPDATE_CPU_DIRTY_LOGGING , vcpu)) +static_call (kvm_x86_update_cpu_dirty_logging)(vcpu); } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
@@ -10755,14 +10758,38 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,return 0 ; } + +static void kvm_mmu_update_cpu_dirty_logging (struct kvm *kvm,bool enable) +{ +struct kvm_arch *ka = &kvm->arch; + +if (!kvm_x86_ops.cpu_dirty_log_size) +return ; + +if ((enable && ++ka->cpu_dirty_logging_count ==1 ) || + (!enable && --ka->cpu_dirty_logging_count ==0 )) +kvm_make_all_cpus_request (kvm,KVM_REQ_UPDATE_CPU_DIRTY_LOGGING ); + +WARN_ON_ONCE (ka->cpu_dirty_logging_count <0 ); +} +static void kvm_mmu_slot_apply_flags (struct kvm *kvm,struct kvm_memory_slot *old,struct kvm_memory_slot *new,enum kvm_mr_change change) { +bool log_dirty_pages = new->flags &KVM_MEM_LOG_DIRTY_PAGES ; +/* -* Nothing to do for RO slots (which can't be dirtied and can't be made -* writable) or CREATE/MOVE/DELETE of a slot. See comments below. +* Update CPU dirty logging if dirty logging is being toggled. This +* applies to all operations. +*/ +if ((old->flags ^ new->flags) &KVM_MEM_LOG_DIRTY_PAGES ) +kvm_mmu_update_cpu_dirty_logging (kvm, log_dirty_pages); + +/* +* Nothing more to do for RO slots (which can't be dirtied and can't be +* made writable) or CREATE/MOVE/DELETE of a slot. See comments below. */ if ((change !=KVM_MR_FLAGS_ONLY ) || (new->flags &KVM_MEM_READONLY ))return ;
@@ -10792,7 +10819,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* MOVE/DELETE: The old mappings will already have been cleaned up by * kvm_arch_flush_shadow_memslot() */ -if (!(new->flags &KVM_MEM_LOG_DIRTY_PAGES )) +if (!log_dirty_pages)kvm_mmu_zap_collapsible_sptes (kvm, new);/*
@@ -10823,7 +10850,7 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* initial-all-set state. Otherwise, depending on whether pml * is enabled the D-bit or the W-bit will be cleared. */ -if (new->flags &KVM_MEM_LOG_DIRTY_PAGES ) { +if (log_dirty_pages) {if (kvm_x86_ops.cpu_dirty_log_size) {if (!kvm_dirty_log_manual_protect_and_init_set (kvm))kvm_mmu_slot_leaf_clear_dirty (kvm, new);
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 12/14 KVM: x86/mmu: Don't set dirty bits when disabling dirty logging w/ PML
In reply to Sean ChristophersonStop setting dirty bits for MMU pages when dirty logging is disabled for a memslot, as PML is now completely disabled when there are no memslots with dirty logging enabled.
This means that spurious PML entries will be created for memslots with dirty logging disabled if at least one other memslot has dirty logging enabled, but for all known use cases, dirty logging is a global VMM control. Furthermore, spurious PML entries are already possible since dirty bits are set only when a dirty logging is turned off, i.e. memslots that are never dirty logged will have dirty bits cleared.
In the end, it's faster overall to eat a few spurious PML entries in the window where dirty logging is being disabled across all memslots.
--- arch/x86/include/asm/kvm_host.h | 2 - arch/x86/kvm/mmu/mmu.c | 45 ----------------- arch/x86/kvm/mmu/tdp_mmu.c | 54 -------------------- arch/x86/kvm/mmu/tdp_mmu.h | 1 - arch/x86/kvm/x86.c | 87 ++++++++++++++------------------- 5 files changed, 36 insertions(+), 153 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -2
@@ -1421,8 +1421,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, struct kvm_memory_slot *memslot);void kvm_mmu_slot_largepage_remove_write_access (struct kvm *kvm,struct kvm_memory_slot *memslot); -void kvm_mmu_slot_set_dirty (struct kvm *kvm, -struct kvm_memory_slot *memslot);void kvm_mmu_zap_all (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen);unsigned long kvm_mmu_calculate_default_mmu_pages (struct kvm *kvm);
arch/x86/kvm/mmu/mmu.c +0 -45
57 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.c +0 -54
60 lines not on this page · raw message
arch/x86/kvm/mmu/tdp_mmu.h +0 -1
@@ -33,7 +33,6 @@ void kvm_tdp_mmu_clear_dirty_pt_masked(struct kvm *kvm,struct kvm_memory_slot *slot, gfn_t gfn, unsigned long mask, bool wrprot); -bool kvm_tdp_mmu_slot_set_dirty (struct kvm *kvm,struct kvm_memory_slot *slot);void kvm_tdp_mmu_zap_collapsible_sptes (struct kvm *kvm,struct kvm_memory_slot *slot);
arch/x86/kvm/x86.c +36 -51
106 lines not on this page · raw message
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Paolo Bonzini
maintainerRe: [PATCH 12/14] KVM: x86/mmu: Don't set dirty bits when disabling dirty logging w/ PML
Quoted 1 line · Sean Christopherson
This means that spurious PML entries will be created for memslots with dirty logging disabled if at least one other memslot has dirty logging enabled, but for all known use cases, dirty logging is a global VMM control.
This is not true. For example QEMU uses dirty logging to track changes to the framebuffer.
However, what you're saying below is true: after a MR_CREATE there will be no shadow pages, and when they are created with mmu_set_spte they will not have the dirty bits set. So there's really no change here for the case of only some memslots having dirty logging enabled. Queued 12 and 13 as well then!
Paolo
Quoted 1 line
Furthermore, spurious PML entries are already possible since dirty bits are set only when a dirty logging is turned off, i.e. memslots that are never dirty logged will have dirty bits cleared.
Sean Christopherson
maintainerPATCH 13/14 KVM: x86: Fold "write-protect large" use case into generic write-protect
In reply to Sean ChristophersonDrop kvm_mmu_slot_largepage_remove_write_access() and refactor its sole caller to use kvm_mmu_slot_remove_write_access(). Remove the now-unused slot_handle_large_level() and slot_handle_all_level() helpers.
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 32 -------------------------------- arch/x86/kvm/x86.c | 32 +++++++++++++++++--------------- 2 files changed, 17 insertions(+), 47 deletions(-)
arch/x86/kvm/mmu/mmu.c +0 -32
@@ -5204,22 +5204,6 @@ slot_handle_level(struct kvm *kvm, struct kvm_memory_slot *memslot, lock_flush_tlb); } -static __always_inline bool -slot_handle_all_level (struct kvm *kvm,struct kvm_memory_slot *memslot, -slot_level_handler fn,bool lock_flush_tlb) -{ -return slot_handle_level (kvm, memslot, fn,PG_LEVEL_4K , -KVM_MAX_HUGEPAGE_LEVEL , lock_flush_tlb); -} - -static __always_inline bool -slot_handle_large_level (struct kvm *kvm,struct kvm_memory_slot *memslot, -slot_level_handler fn,bool lock_flush_tlb) -{ -return slot_handle_level (kvm, memslot, fn,PG_LEVEL_4K +1 , -KVM_MAX_HUGEPAGE_LEVEL , lock_flush_tlb); -} - static __always_inline boolslot_handle_leaf (struct kvm *kvm,struct kvm_memory_slot *memslot,slot_level_handler fn,bool lock_flush_tlb)
@@ -5584,22 +5568,6 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,kvm_arch_flush_remote_tlbs_memslot (kvm, memslot); } -void kvm_mmu_slot_largepage_remove_write_access (struct kvm *kvm, -struct kvm_memory_slot *memslot) -{ -bool flush; - -write_lock (&kvm->mmu_lock); - flush =slot_handle_large_level (kvm, memslot, slot_rmap_write_protect, - false); -if (is_tdp_mmu_enabled (kvm)) - flush |=kvm_tdp_mmu_wrprot_slot (kvm, memslot,PG_LEVEL_2M ); -write_unlock (&kvm->mmu_lock); - -if (flush) -kvm_arch_flush_remote_tlbs_memslot (kvm, memslot); -} -void kvm_mmu_zap_all (struct kvm *kvm) {struct kvm_mmu_page *sp, *node;
arch/x86/kvm/x86.c +17 -15
@@ -10829,24 +10829,25 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,*/ kvm_mmu_zap_collapsible_sptes (kvm, new); }else { -/* -* Large sptes are write-protected so they can be split on first -* write. New large sptes cannot be created for this slot until -* the end of the logging. See the comments in fast_page_fault(). -* -* For small sptes, nothing is done if the dirty log is in the -* initial-all-set state. Otherwise, depending on whether pml -* is enabled the D-bit or the W-bit will be cleared. -*/ +/* By default, write-protect everything to log writes. */ +int level =PG_LEVEL_4K ; +if (kvm_x86_ops.cpu_dirty_log_size) { +/* +* Clear all dirty bits, unless pages are treated as +* dirty from the get-go. +*/ if (!kvm_dirty_log_manual_protect_and_init_set (kvm))kvm_mmu_slot_leaf_clear_dirty (kvm, new); -kvm_mmu_slot_largepage_remove_write_access (kvm, new); - }else { -int level = -kvm_dirty_log_manual_protect_and_init_set (kvm) ? -PG_LEVEL_2M :PG_LEVEL_4K ; +/* +* Write-protect large pages on write so that dirty +* logging happens at 4k granularity. No need to +* write-protect small SPTEs since write accesses are +* logged by the CPU via dirty bits. +*/ + level =PG_LEVEL_2M ; + }else if (kvm_dirty_log_manual_protect_and_init_set (kvm)) {/* * If we're with initial-all-set, we don't need * to write protect any small page because
@@ -10855,8 +10856,9 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm,* so that the page split can happen lazily on * the first write to the huge page. */ -kvm_mmu_slot_remove_write_access (kvm, new, level); + level =PG_LEVEL_2M ; } +kvm_mmu_slot_remove_write_access (kvm, new, level); } }
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerPATCH 14/14 KVM: x86/mmu: Remove a variety of unnecessary exports
In reply to Sean ChristophersonRemove several exports from the MMU that are no longer necessary.
No functional change intended.
--- arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/mmu/mmu.c | 35 ++++++++++++++------------------- 2 files changed, 15 insertions(+), 21 deletions(-)
arch/x86/include/asm/kvm_host.h +0 -1
@@ -1592,7 +1592,6 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu);void kvm_update_dr7 (struct kvm_vcpu *vcpu);int kvm_mmu_unprotect_page (struct kvm *kvm,gfn_t gfn); -int kvm_mmu_unprotect_page_virt (struct kvm_vcpu *vcpu,gva_t gva);void __kvm_mmu_free_some_pages (struct kvm_vcpu *vcpu);int kvm_mmu_load (struct kvm_vcpu *vcpu);void kvm_mmu_unload (struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/mmu.c +15 -20
@@ -2466,7 +2466,21 @@ int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn)return r; } -EXPORT_SYMBOL_GPL (kvm_mmu_unprotect_page); + +static int kvm_mmu_unprotect_page_virt (struct kvm_vcpu *vcpu,gva_t gva) +{ +gpa_t gpa; +int r; + +if (vcpu->arch.mmu->direct_map) +return 0 ; + + gpa =kvm_mmu_gva_to_gpa_read (vcpu, gva,NULL ); + + r =kvm_mmu_unprotect_page (vcpu->kvm, gpa >>PAGE_SHIFT ); + +return r; +}static void kvm_unsync_page (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp) {
@@ -3411,7 +3425,6 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)kvm_mmu_audit (vcpu,AUDIT_POST_SYNC );write_unlock (&vcpu->kvm->mmu_lock); } -EXPORT_SYMBOL_GPL (kvm_mmu_sync_roots);static gpa_t nonpaging_gva_to_gpa (struct kvm_vcpu *vcpu,gpa_t vaddr,u32 access,struct x86_exception *exception)
@@ -4977,22 +4990,6 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,write_unlock (&vcpu->kvm->mmu_lock); } -int kvm_mmu_unprotect_page_virt (struct kvm_vcpu *vcpu,gva_t gva) -{ -gpa_t gpa; -int r; - -if (vcpu->arch.mmu->direct_map) -return 0 ; - - gpa =kvm_mmu_gva_to_gpa_read (vcpu, gva,NULL ); - - r =kvm_mmu_unprotect_page (vcpu->kvm, gpa >>PAGE_SHIFT ); - -return r; -} -EXPORT_SYMBOL_GPL (kvm_mmu_unprotect_page_virt); -int kvm_mmu_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa,u64 error_code,void *insn,int insn_len) {
@@ -5091,7 +5088,6 @@ void kvm_mmu_invalidate_gva(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, mmu->invlpg (vcpu, gva, root_hpa); } } -EXPORT_SYMBOL_GPL (kvm_mmu_invalidate_gva);void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva) {
@@ -5131,7 +5127,6 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid)* for them. */ } -EXPORT_SYMBOL_GPL (kvm_mmu_invpcid_gva);void kvm_configure_mmu (bool enable_tdp,int tdp_max_root_level,int tdp_huge_page_level)
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
Sean Christopherson
maintainerQuoted 5 lines · Sean Christopherson
Paolo, this is more or less ready, but on final read-through before sending I realized it would be a good idea to WARN during VM destruction if
cpu_dirty_logging_countis non-zero. I wanted to get you this before the 5.12 window opens in case you want the TDP MMU fixes for 5.12. I'll do the above change and retest next week (note, Monday is a US holiday).
Verified cpu_dirty_logging_count does indeed hit zero during VM destruction.
Adding a WARN to KVM would require adding an arch hook to kvm_free_memslots(), otherwise the count will be non-zero if the VM is destroyed with dirty logging active. That doesn't seem worthwhile, so I'm not planning on pursuing a WARN for the upstream code.
Quoted 68 lines · Sean Christopherson
Paolo, this is more or less ready, but on final read-through before sending I realized it would be a good idea to WARN during VM destruction if
cpu_dirty_logging_countis non-zero. I wanted to get you this before the 5.12 window opens in case you want the TDP MMU fixes for 5.12. I'll do the above change and retest next week (note, Monday is a US holiday).On to the code...
This started out as a small tweak to collapsible SPTE zapping in the TDP MMU, and ended up as a rather large overhaul of CPU dirty logging, a.k.a. PML.
Four main highlights:
- Do a more precise check on whether or not a SPTE should be zapped to rebuild it as a large page.
- Disable PML when running L2. PML is fully emulated for L1 VMMs, thus enabling PML in L2 can only hurt and never help.
- Drop the existing PMLkvm_x86_ops. They're basically trampolines into the MMU, and IMO do far more harm than good.
- Turn on PML only when it's needed instead of setting all dirty bits to soft disable PML.What led me down the rabbit's hole of ripping out the existing PML
kvm_x86_opsisn't really shown here. Prior to incorporating Makarand's patch, which allowed for the wholesale remove of setting dirty bits, I spent a bunch of time poking around the "set dirty bits" code. My original changes optimized that path to skip setting dirty bits in the nested MMU, since the nested MMU relies on write-protection and not PML. That in turn allowed the TDP MMU zapping to completely skip walking the rmaps, but doing so based on a bunch of callbacks was a twisted mess.Happily, those patches got dropped in favor of nuking the code entirely.
Ran selftest and unit tests, and migrated actual VMs on AMD and Intel, with and without TDP MMU, and with and without EPT. The AMD system I'm testing on infinite loops on the reset vector due to a #PF when NPT is disabled, so that didn't get tested. That reproduces with kvm/next, I'll dig into it next week (no idea if it's a KVM or hardware issue).
For actual migration, I ran kvm-unit-tests in L1 along with stress to hammer memory, and verified migration was effectively blocked until the stress threads were killed (I didn't feel like figuring out how to throttle the VM).
Makarand Sonare (1):
KVM: VMX: Dynamically enable/disable PML based on memslot dirty
loggingSean Christopherson (13):
KVM: x86/mmu: Expand collapsible SPTE zap for TDP MMU toZONE_DEVICE
pages
KVM: x86/mmu: Don't unnecessarily write-protect small pages in TDP MMU
KVM: x86/mmu: Split out max mapping level calculation to helper
KVM: x86/mmu: Pass the memslot to the rmap callbacks
KVM: x86/mmu: Consult max mapping level when zapping collapsible SPTEs
KVM: nVMX: Disable PML in hardware when running L2
KVM: x86/mmu: Expand on the comment in
kvm_vcpu_ad_need_write_protect()
KVM: x86/mmu: Make dirty log size hook (PML) a value, not a function
KVM: x86: Move MMU's PML logic to common code
KVM: x86: Further clarify the logic and comments for toggling log
dirty
KVM: x86/mmu: Don't set dirty bits when disabling dirty logging w/ PML
KVM: x86: Fold "write-protect large" use case into generic
write-protect
KVM: x86/mmu: Remove a variety of unnecessary exports
Quoted 12 lines
arch/x86/include/asm/kvm-x86-ops.h | 6 +- arch/x86/include/asm/kvm_host.h | 36 +---- arch/x86/kvm/mmu/mmu.c | 203 +++++++++-------------------- arch/x86/kvm/mmu/mmu_internal.h | 7 +- arch/x86/kvm/mmu/tdp_mmu.c | 66 +--------- arch/x86/kvm/mmu/tdp_mmu.h | 3 +- arch/x86/kvm/vmx/nested.c | 34 +++-- arch/x86/kvm/vmx/vmx.c | 94 +++++-------- arch/x86/kvm/vmx/vmx.h | 2 + arch/x86/kvm/x86.c | 145 +++++++++++++-------- 10 files changed, 230 insertions(+), 366 deletions(-)
Queued 1-9 and 14 until you clarify my doubt about patch 10, thanks.
Paolo