Skip to contentSkip to conversation

KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled

PATCH Superseded David Matlack 3 participants 11 messages Oct 27, 2022 – Oct 28, 2022

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled

This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.

Tested manually and ran all selftests.

David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabled

 arch/x86/kvm/mmu/mmu.c   |  8 ++++++++
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 10 ++++++++++
 3 files changed, 20 insertions(+)

base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354

Signature · 1 line
2.38.1.273.g43a17bfeac-goog
#1 lore raw

PATCH 1/2 KVM: Keep track of the number of memslots with dirty logging enabled

Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 10 ++++++++++
 2 files changed, 12 insertions(+)
include/linux/kvm_host.h +2 -0
index 32f259fa5801..25ed8c1725ff 100644
@@ -709,6 +709,8 @@ struct kvm {
 	struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM][2];
 	/* The current active memslot set for each address space */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
+	/* The number of memslots with dirty logging enabled. */
+	int nr_memslots_dirty_logging;
 	struct xarray vcpu_array;
 
 	/* Used to wait for completion of MMU notifiers.  */
virt/kvm/kvm_main.c +10 -0
index e30f1b4ecfa5..57e4406005cd 100644
@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 				     const struct kvm_memory_slot *new,
 				     enum kvm_mr_change change)
 {
+	int old_flags = old ? old->flags : 0;
+	int new_flags = new ? new->flags : 0;
+
 	/*
 	 * Update the total number of memslot pages before calling the arch
 	 * hook so that architectures can consume the result directly.
@@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 	else if (change == KVM_MR_CREATE)
 		kvm->nr_memslot_pages += new->npages;
 
+	if ((old_flags ^ new_flags) & KVM_MEM_LOG_DIRTY_PAGES) {
+		if (new_flags & KVM_MEM_LOG_DIRTY_PAGES)
+			kvm->nr_memslots_dirty_logging++;
+		else
+			kvm->nr_memslots_dirty_logging--;
+	}
+
 	kvm_arch_commit_memory_region(kvm, old, new, change);
 
 	switch (change) {
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
#5 lore raw

PATCH 2/2 KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is enabled

In reply to David Matlack

Do not recover NX Huge Pages if dirty logging is enabled on any memslot. Zapping a region that is being dirty tracked is a waste of CPU cycles (both by the recovery worker, and subsequent vCPU faults) since the memory will just be faulted back in at the same 4KiB granularity.

Use kvm->nr_memslots_dirty_logging as a cheap way to check if NX Huge Pages are being dirty tracked. This has the additional benefit of ensuring that the NX recovery worker uses little-to-no CPU during the precopy phase of a live migration.

Note, kvm->nr_memslots_dirty_logging can result in false positives and false negatives, e.g. if dirty logging is only enabled on a subset of memslots or the recovery worker races with a memslot update. However there are no correctness issues either way, and eventually NX Huge Pages will be recovered once dirty logging is disabled on all memslots.

An alternative approach would be to lookup the memslot of each NX Huge Page and check if it is being dirty tracked. However, this would increase the CPU usage of the recovery worker and MMU lock hold time in write mode, especially in VMs with a large number of memslots.

---
 arch/x86/kvm/mmu/mmu.c | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/kvm/mmu/mmu.c +8 -0
index 6f81539061d6..b499d3757173 100644
@@ -6806,6 +6806,14 @@ static void kvm_recover_nx_lpages(struct kvm *kvm)
 	bool flush = false;
 	ulong to_zap;
 
+	/*
+	 * Do not attempt to recover NX Huge Pages while dirty logging is
+	 * enabled since any subsequent accesses by a vCPUs will just fault the
+	 * memory back in at the same 4KiB granularity.
+	 */
+	if (READ_ONCE(kvm->nr_memslots_dirty_logging))
+		return;
+
 	rcu_idx = srcu_read_lock(&kvm->srcu);
 	write_lock(&kvm->mmu_lock);
 
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
#2 lore raw

Re: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled

In reply to David Matlack
Quoted 8 lines · David Matlack

Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 10 ++++++++++

Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.

include/linux/kvm_host.h struct kvm { · line 713

Quoted 12 lines
 2 files changed, 12 insertions(+)
include/linux/kvm_host.h +2 -0
index 32f259fa5801..25ed8c1725ff 100644
@@ -709,6 +709,8 @@ struct kvm {
 	struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM][2];
 	/* The current active memslot set for each address space */
 	struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
+	/* The number of memslots with dirty logging enabled. */
+	int nr_memslots_dirty_logging;

I believe this can technically be a u16, as even with SMM KVM ensures the total number of memslots fits in a u16. A BUILD_BUG_ON() sanity check is probably a good idea regardless.

virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1645

Quoted 13 lines
include/linux/kvm_host.h +0 -0
 	struct xarray vcpu_array;
 
 	/* Used to wait for completion of MMU notifiers.  */
virt/kvm/kvm_main.c +2 -0
index e30f1b4ecfa5..57e4406005cd 100644
@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 				     const struct kvm_memory_slot *new,
 				     enum kvm_mr_change change)
 {
+	int old_flags = old ? old->flags : 0;
+	int new_flags = new ? new->flags : 0;

Not that it really matters, but kvm_memory_slot.flags is a u32.

virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1660

Quoted 12 lines
virt/kvm/kvm_main.c +5 -0
 	/*
 	 * Update the total number of memslot pages before calling the arch
 	 * hook so that architectures can consume the result directly.
@@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,
 	else if (change == KVM_MR_CREATE)
 		kvm->nr_memslot_pages += new->npages;
 
+	if ((old_flags ^ new_flags) & KVM_MEM_LOG_DIRTY_PAGES) {
+		if (new_flags & KVM_MEM_LOG_DIRTY_PAGES)
+			kvm->nr_memslots_dirty_logging++;
+		else
+			kvm->nr_memslots_dirty_logging--;

A sanity check that KVM hasn't botched the count is probably a good idea. E.g. __kvm_set_memory_region() as a WARN_ON_ONCE() sanity check that KVM won't end up underflowing nr_memslot_pages.

Quoted 8 lines
virt/kvm/kvm_main.c +2 -0
+	}
+
 	kvm_arch_commit_memory_region(kvm, old, new, change);
 
 	switch (change) {
Signature · 1 line
2.38.1.273.g43a17bfeac-goog
#3 lore raw

Re: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled

Quoted 6 lines · Sean Christopherson
Quoted 4 lines · David Matlack

Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.

Quoted 8 lines
Quoted 4 lines
---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 10 ++++++++++

Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.

I agree, but what is the downside of putting it in common code? The downside of putting it in architecture-specific code is if any other architecture needs it (or something similar) in the future they are unlikely to look through x86 code to see if it already exists. i.e. we're more likely to end up with duplicate code.

And while the count is not stable outside of slots_lock, it could still theoretically be used under slots_lock to coordinate something that depends on dirty logging being enabled in any slot. In our internal kernel, for example, we use it to decide when to create/destroy the KVM dirty log worker threads (although I doubt that specific usecase will ever see the light of day upstream :).

Quoted 17 lines

include/linux/kvm_host.h struct kvm { · line 713

Quoted 13 lines
 2 files changed, 12 insertions(+)
include/linux/kvm_host.h +2 -0
index 32f259fa5801..25ed8c1725ff 100644
@@ -709,6 +709,8 @@ struct kvm {
      struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM][2];
      /* The current active memslot set for each address space */
      struct kvm_memslots __rcu *memslots[KVM_ADDRESS_SPACE_NUM];
+     /* The number of memslots with dirty logging enabled. */
+     int nr_memslots_dirty_logging;

I believe this can technically be a u16, as even with SMM KVM ensures the total number of memslots fits in a u16. A BUILD_BUG_ON() sanity check is probably a good idea regardless.

Will do.

Quoted 16 lines
Quoted 9 lines
include/linux/kvm_host.h +0 -0
      struct xarray vcpu_array;
 
      /* Used to wait for completion of MMU notifiers.  */
virt/kvm/kvm_main.c +0 -0
index e30f1b4ecfa5..57e4406005cd 100644

virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1645

Quoted 5 lines
virt/kvm/kvm_main.c +2 -0
@@ -1641,6 +1641,9 @@ static void kvm_commit_memory_region(struct kvm *kvm,
                                   const struct kvm_memory_slot *new,
                                   enum kvm_mr_change change)
 {
+     int old_flags = old ? old->flags : 0;
+     int new_flags = new ? new->flags : 0;

Not that it really matters, but kvm_memory_slot.flags is a u32.

Oops, thanks.

Quoted 8 lines
Quoted 8 lines
virt/kvm/kvm_main.c +0 -0
      /*
       * Update the total number of memslot pages before calling the arch
       * hook so that architectures can consume the result directly.
@@ -1650,6 +1653,13 @@ static void kvm_commit_memory_region(struct kvm *kvm,
      else if (change == KVM_MR_CREATE)
              kvm->nr_memslot_pages += new->npages;
 
Quoted 9 lines

virt/kvm/kvm_main.c static void kvm_commit_memory_region(struct kvm *kvm, · line 1660

Quoted 5 lines
virt/kvm/kvm_main.c +5 -0
+     if ((old_flags ^ new_flags) & KVM_MEM_LOG_DIRTY_PAGES) {
+             if (new_flags & KVM_MEM_LOG_DIRTY_PAGES)
+                     kvm->nr_memslots_dirty_logging++;
+             else
+                     kvm->nr_memslots_dirty_logging--;

A sanity check that KVM hasn't botched the count is probably a good idea. E.g. __kvm_set_memory_region() as a WARN_ON_ONCE() sanity check that KVM won't end up underflowing nr_memslot_pages.

Good idea, will do.

#4 lore raw

Re: [PATCH 1/2] KVM: Keep track of the number of memslots with dirty logging enabled

Quoted 12 lines · David Matlack
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · David Matlack

Add a new field to struct kvm that keeps track of the number of memslots with dirty logging enabled. This will be used in a future commit to cheaply check if any memslot is doing dirty logging.

---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/kvm_main.c      | 10 ++++++++++
Quoted 5 lines
Quoted 3 lines

Why put this in common code? I'm having a hard time coming up with a second use case since the count isn't stable, i.e. it can't be used for anything except scenarios like x86's NX huge page mitigation where a false negative/positive is benign.

I agree, but what is the downside of putting it in common code?

The potential for misuse, e.g. outside of slots_lock.

Quoted 11 lines

The downside of putting it in architecture-specific code is if any other architecture needs it (or something similar) in the future they are unlikely to look through x86 code to see if it already exists. i.e. we're more likely to end up with duplicate code.

And while the count is not stable outside of slots_lock, it could still theoretically be used under slots_lock to coordinate something that depends on dirty logging being enabled in any slot. In our internal kernel, for example, we use it to decide when to create/destroy the KVM dirty log worker threads (although I doubt that specific usecase will ever see the light of day upstream :).

Yeah, I'm definitely not dead set against putting it in common code. I suspect I'm a little overly sensitive at the moment to x86 pushing a bunch of x86-centric logic into kvm_main.c and making life miserable for everyone. Been spending far too much time unwinding the mess that is kvm_init()...

maintainer
#6 lore raw
In reply to David Matlack
Quoted 12 lines · David Matlack

This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.

Tested manually and ran all selftests.

David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabled

Quoted 7 lines
  arch/x86/kvm/mmu/mmu.c   |  8 ++++++++
  include/linux/kvm_host.h |  2 ++
  virt/kvm/kvm_main.c      | 10 ++++++++++
  3 files changed, 20 insertions(+)

base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354

This can be a bit problematic because for example you could have dirty logging enabled only for a framebuffer or similar. In this case the memory being logged will not be the same as the one that is NX-split.

Perhaps we can take advantage of eager page splitting, that is you can add a bool to kvm_mmu_page that is set by shadow_mmu_get_sp_for_split and tdp_mmu_alloc_sp_for_split (or a similar place)?

Paolo

#7 lore raw
Quoted 20 lines · Paolo Bonzini
Quoted 18 lines · David Matlack

This series turns off the NX Huge Page recovery worker when any memslot has dirty logging enabled. This avoids theoretical performance problems and reduces the CPU usage of NX Huge Pages when a VM is in the pre-copy phase of a Live Migration.

Tested manually and ran all selftests.

David Matlack (2):
KVM: Keep track of the number of memslots with dirty logging enabled
KVM: x86/mmu: Do not recover NX Huge Pages when dirty logging is
enabled

  arch/x86/kvm/mmu/mmu.c   |  8 ++++++++
  include/linux/kvm_host.h |  2 ++
  virt/kvm/kvm_main.c      | 10 ++++++++++
  3 files changed, 20 insertions(+)
Quoted 5 lines
Quoted 1 line

base-commit: e18d6152ff0f41b7f01f9817372022df04e0d354

This can be a bit problematic because for example you could have dirty logging enabled only for a framebuffer or similar. In this case the memory being logged will not be the same as the one that is NX-split.

Ah, thanks for pointing that out. It's good to know there are other use-cases for dirty logging outside of live migration. At Google we tend to hyper focus on LM.

Quoted 4 lines

Perhaps we can take advantage of eager page splitting, that is you can add a bool to kvm_mmu_page that is set by shadow_mmu_get_sp_for_split and tdp_mmu_alloc_sp_for_split (or a similar place)?

I don't think that would help since eagerly-split pages aren't on the list of possible NX Huge Pages. i.e. the recovery worker won't try to zap them. The main thing I want to avoid is the recovery worker zapping ranges that were actually split for NX Huge Pages while they are being dirty tracked.

I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.

Quoted 3 lines

Paolo

#8 lore raw
Quoted 4 lines · David Matlack

I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.

If you end up grabbing the memslot, use kvm_mmu_max_mapping_level() instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.

#9 lore raw
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack

I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.

Quoted 5 lines

If you end up grabbing the memslot, use kvm_mmu_max_mapping_level() instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.

kvm_mmu_max_mapping_level() doesn't check if dirty logging is enabled and does the unnecessary work of checking the host mapping level (which requires knowing the pfn). I could refactor kvm_mmu_hugepage_adjust() and kvm_mmu_max_mapping_level() though to achieve what you suggest. Specifically, when recovering NX Huge Pages, check if dirty logging is enabled and if a huge page is disallowed (lpage_info_slot), and share that code with the fault handler.

#10 lore raw
Quoted 8 lines · David Matlack
Quoted 7 lines · Sean Christopherson
Quoted 4 lines · David Matlack

I'll experiment with a more accurate solution. i.e. have the recovery worker lookup the memslot for each SP and check if it has dirty logging enabled. Maybe the increase in CPU usage won't be as bad as I thought.

Quoted 7 lines
Quoted 5 lines

If you end up grabbing the memslot, use kvm_mmu_max_mapping_level() instead of checking only dirty logging. The way KVM will avoid zapping shadow pages that could have been NX huge pages when they were created, but can no longer be NX huge pages due to something other than dirty logging, e.g. because the gfn is being shadow for nested TDP.

kvm_mmu_max_mapping_level() doesn't check if dirty logging is enabled

Gah, I forgot that kvm_mmu_hugepage_adjust() does a one-off check for dirty logging instead of the info being fed into slot->arch.lpage_info. Never mind.