KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled
cover KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled
This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.
Tested manually and ran all selftests.
David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabled
arch/x86/kvm/mmu/mmu.c | 8 ++++++++ include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++ 3 files changed, 20 insertions(+)
base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
PATCH 1/2 KVM: Keep track of the number of memslots with dirty logging enabled
Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.
--- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++ 2 files changed, 12 insertions(+)
include/linux/kvm_host.h +2 -0
@@ -709,6 +709,8 @@ struct kvm {struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM ][2 ];/* The current active memslot set for each address space */ struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ]; +/* The number of memslots with dirty logging enabled. */ +int nr_memslots_dirty_logging;struct xarray vcpu_array;/* Used to wait for completion of MMU notifiers. */
virt/kvm/kvm_main.c +10 -0
@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { + int old_flags = old ? old->flags : 0; +int new_flags = new ? new->flags :0 ; +/* * Update the total number of memslot pages before calling the arch * hook so that architectures can consume the result directly.
@@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,else if (change ==KVM_MR_CREATE ) kvm->nr_memslot_pages += new->npages; +if ((old_flags ^ new_flags) &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new_flags &KVM_MEM_LOG_DIRTY_PAGES ) + kvm->nr_memslots_dirty_logging++; +else + kvm->nr_memslots_dirty_logging--; + } +kvm_arch_commit_memory_region (kvm, old, new, change);switch (change) {
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
PATCH 2/2 KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled
In reply to David MatlackDo not recover NX Huge Pages if dirty logging is enabled on any memslot. Zapping a region that is being dirty tracked is a waste of CPU cycles (both by the recovery worker, and subsequent vCPU faults) since the memory will just be faulted back in at the same 4KiB granularity.
Use kvm->nr_memslots_dirty_logging as a cheap way to check if NX Huge Pages are being dirty tracked. This has the additional benefit of ensuring that the NX recovery worker uses little-to-no CPU during the precopy phase of a live migration.
Note, kvm->nr_memslots_dirty_logging can result in false positives and false negatives, e.g. if dirty logging is only enabled on a subset of memslots or the recovery worker races with a memslot update. However there are no correctness issues either way, and eventually NX Huge Pages will be recovered once dirty logging is disabled on all memslots.
An alternative approach would be to lookup the memslot of each NX Huge Page and check if it is being dirty tracked. However, this would increase the CPU usage of the recovery worker and MMU lock hold time in write mode, especially in VMs with a large number of memslots.
--- arch/x86/kvm/mmu/mmu.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kvm/mmu/mmu.c +8 -0
@@ -6806,6 +6806,14 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)bool flush = false;ulong to_zap; +/* +* Do not attempt to recover NX Huge Pages while dirty logging is +* enabled since any subsequent accesses by a vCPUs will just fault the +* memory back in at the same 4KiB granularity. +*/ +if (READ_ONCE (kvm->nr_memslots_dirty_logging)) +return ; + rcu_idx =srcu_read_lock (&kvm->srcu);write_lock (&kvm->mmu_lock);
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
Sean Christopherson
maintainerRe: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled
In reply to David MatlackQuoted 8 lines · David Matlack
Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.
Signed-off-by David Matlack <[email protected]>--- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++
Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.
include/linux/kvm_host.h struct kvm { · line 713
Quoted 12 lines
2 files changed, 12 insertions(+)include/linux/kvm_host.h +2
-0@@ -709,6 +709,8 @@ struct kvm {struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM ][2 ];/* The current active memslot set for each address space */ struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ]; +/* The number of memslots with dirty logging enabled. */ +int nr_memslots_dirty_logging;
I believe this can technically be a u16, as even with SMM KVM ensures the total number of memslots fits in a u16. A BUILD_BUG_ON() sanity check is probably a good idea regardless.
virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1645
Quoted 13 lines
include/linux/kvm_host.h +0
-0struct xarray vcpu_array;/* Used to wait for completion of MMU notifiers. */ virt/kvm/kvm_main.c +2
-0@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { + int old_flags = old ? old->flags : 0; +int new_flags = new ? new->flags :0 ;
Not that it really matters, but kvm_memory_slot.flags is a u32.
virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1660
Quoted 12 lines
virt/kvm/kvm_main.c +5
-0/* * Update the total number of memslot pages before calling the arch * hook so that architectures can consume the result directly. @@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,else if (change ==KVM_MR_CREATE ) kvm->nr_memslot_pages += new->npages; +if ((old_flags ^ new_flags) &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new_flags &KVM_MEM_LOG_DIRTY_PAGES ) + kvm->nr_memslots_dirty_logging++; +else + kvm->nr_memslots_dirty_logging--;
A sanity check that KVM hasn't botched the count is probably a good idea. E.g. __kvm_set_memory_region() as a WARN_ON_ONCE() sanity check that KVM won't end up underflowing nr_memslot_pages.
Quoted 8 lines
virt/kvm/kvm_main.c +2
-0+ } +kvm_arch_commit_memory_region (kvm, old, new, change);switch (change) {Signature · 1 line
2.38.1.273.g43a17bfeac-goog
Re: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled
Quoted 6 lines · Sean Christopherson
Quoted 4 lines · David Matlack
Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.
Quoted 8 lines
Quoted 4 lines
Signed-off-by David Matlack <[email protected]>--- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.
I agree, but what is the downside of putting it in common code? The downside of putting it in architecture-specific code is if any other architecture needs it (or something similar) in the future they are unlikely to look through x86 code to see if it already exists. i.e. we're more likely to end up with duplicate code.
And while the count is not stable outside of slots_lock, it could still theoretically be used under slots_lock to coordinate something that depends on dirty logging being enabled in any slot. In our internal kernel, for example, we use it to decide when to create/destroy the KVM dirty log worker threads (although I doubt that specific usecase will ever see the light of day upstream :).
Quoted 17 lines
include/linux/kvm_host.h struct kvm { · line 713
Quoted 13 lines
2 files changed, 12 insertions(+)include/linux/kvm_host.h +2
-0@@ -709,6 +709,8 @@ struct kvm {struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM ][2 ];/* The current active memslot set for each address space */ struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM ]; +/* The number of memslots with dirty logging enabled. */ +int nr_memslots_dirty_logging;I believe this can technically be a u16, as even with SMM KVM ensures the total number of memslots fits in a u16. A
BUILD_BUG_ON()sanity check is probably a good idea regardless.
Will do.
Quoted 16 lines
Quoted 9 lines
include/linux/kvm_host.h +0
-0struct xarray vcpu_array;/* Used to wait for completion of MMU notifiers. */ virt/kvm/kvm_main.c +0
-0virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1645
Quoted 5 lines
virt/kvm/kvm_main.c +2
-0@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,const struct kvm_memory_slot *new, enum kvm_mr_change change) { + int old_flags = old ? old->flags : 0; +int new_flags = new ? new->flags :0 ;Not that it really matters, but kvm_memory_slot.flags is a u32.
Oops, thanks.
Quoted 8 lines
Quoted 8 lines
virt/kvm/kvm_main.c +0
-0/* * Update the total number of memslot pages before calling the arch * hook so that architectures can consume the result directly. @@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,else if (change ==KVM_MR_CREATE ) kvm->nr_memslot_pages += new->npages;
Quoted 9 lines
virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1660
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+if ((old_flags ^ new_flags) &KVM_MEM_LOG_DIRTY_PAGES ) { +if (new_flags &KVM_MEM_LOG_DIRTY_PAGES ) + kvm->nr_memslots_dirty_logging++; +else + kvm->nr_memslots_dirty_logging--;A sanity check that KVM hasn't botched the count is probably a good idea. E.g.
__kvm_set_memory_region()as aWARN_ON_ONCE()sanity check that KVM won't end up underflowingnr_memslot_pages.
Good idea, will do.
Sean Christopherson
maintainerRe: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled
Quoted 12 lines · David Matlack
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · David Matlack
Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.
Signed-off-by David Matlack <[email protected]>--- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++
Quoted 5 lines
Quoted 3 lines
Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.
I agree, but what is the downside of putting it in common code?
The potential for misuse, e.g. outside of slots_lock.
Quoted 11 lines
The downside of putting it in architecture-specific code is if any other architecture needs it (or something similar) in the future they are unlikely to look through x86 code to see if it already exists. i.e. we're more likely to end up with duplicate code.
And while the count is not stable outside of
slots_lock, it could still theoretically be used underslots_lockto coordinate something that depends on dirty logging being enabled in any slot. In our internal kernel, for example, we use it to decide when to create/destroy the KVM dirty log worker threads (although I doubt that specific usecase will ever see the light of day upstream :).
Yeah, I'm definitely not dead set against putting it in common code. I suspect I'm a little overly sensitive at the moment to x86 pushing a bunch of x86-centric logic into kvm_main.c and making life miserable for everyone. Been spending far too much time unwinding the mess that is kvm_init()...
Paolo Bonzini
maintainerQuoted 12 lines · David Matlack
This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.
Tested manually and ran all selftests.
David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabled
Quoted 7 lines
arch/x86/kvm/mmu/mmu.c | 8 ++++++++ include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++ 3 files changed, 20 insertions(+)base-commit:
e18d6152ff0f41b7f01f9817372022df04e0d354
This can be a bit problematic because for example you could have dirty logging enabled only for a framebuffer or similar. In this case the memory being logged will not be the same as the one that is NX-split.
Perhaps we can take advantage of eager page splitting, that is you can add a bool to kvm_mmu_page that is set by shadow_mmu_get_sp_for_split and tdp_mmu_alloc_sp_for_split (or a similar place)?
Paolo
Quoted 20 lines · Paolo Bonzini
Quoted 18 lines · David Matlack
This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.
Tested manually and ran all selftests.
David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabledarch/x86/kvm/mmu/mmu.c | 8 ++++++++ include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 10 ++++++++++ 3 files changed, 20 insertions(+)
Quoted 5 lines
Quoted 1 line
base-commit:
e18d6152ff0f41b7f01f9817372022df04e0d354This can be a bit problematic because for example you could have dirty logging enabled only for a framebuffer or similar. In this case the memory being logged will not be the same as the one that is NX-split.
Ah, thanks for pointing that out. It's good to know there are other use-cases for dirty logging outside of live migration. At Google we tend to hyper focus on LM.
Quoted 4 lines
Perhaps we can take advantage of eager page splitting, that is you can add a bool to
kvm_mmu_pagethat is set byshadow_mmu_get_sp_for_splitandtdp_mmu_alloc_sp_for_split(or a similar place)?
I don't think that would help since eagerly-split pages aren't on the list of possible NX Huge Pages. i.e. the recovery worker won't try to zap them. The main thing I want to avoid is the recovery worker zapping ranges that were actually split for NX Huge Pages while they are being dirty tracked.
I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.
Quoted 3 lines
Paolo
Sean Christopherson
maintainerQuoted 4 lines · David Matlack
I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.
If you end up grabbing the memslot, use kvm_mmu_max_mapping_level() instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.
Quoted 5 lines
If you end up grabbing the memslot, use
kvm_mmu_max_mapping_level()instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.
kvm_mmu_max_mapping_level() doesn't check if dirty logging is enabled and does the unnecessary work of checking the host mapping level (which requires knowing the pfn). I could refactor kvm_mmu_hugepage_adjust() and kvm_mmu_max_mapping_level() though to achieve what you suggest. Specifically, when recovering NX Huge Pages, check if dirty logging is enabled and if a huge page is disallowed (lpage_info_slot), and share that code with the fault handler.
Sean Christopherson
maintainerQuoted 8 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack
I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.
Quoted 7 lines
Quoted 5 lines
If you end up grabbing the memslot, use
kvm_mmu_max_mapping_level()instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.
kvm_mmu_max_mapping_level()doesn't check if dirty logging is enabled
Gah, I forgot that kvm_mmu_hugepage_adjust() does a one-off check for dirty logging instead of the info being fed into slot->arch.lpage_info. Never mind.