Skip to contentSkip to conversation

KVM: VMX: Enable/disable PML when dirty logging gets enabled/disabled

PATCH RESEND In review Makarand Sonare 3 participants 12 messages Feb 10, 2021 – Feb 12, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH RESEND 1/1 KVM: VMX: Enable/disable PML when dirty logging gets enabled/disabled

Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Clearing the dirty bits is an expensive operation which can cause severe MMU lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration. Also, this would break if some other code path clears the dirty bits in which case, PML will actually start logging dirty pages even when dirty logging is disabled incurring unnecessary vmexits when the PML buffer becomes full. In order to avoid this extra overhead, we should enable or disable PML in VMCS when dirty logging gets enabled or disabled instead of keeping it always enabled.

Tested:
kvm-unit-tests
dirty_log_test
dirty_log_perf_test

---
 arch/x86/include/asm/kvm_host.h |  4 +++
 arch/x86/kvm/vmx/nested.c       |  5 ++++
 arch/x86/kvm/vmx/vmx.c          | 49 +++++++++++++++++++++++++++++++--
 arch/x86/kvm/vmx/vmx.h          |  3 ++
 arch/x86/kvm/x86.c              |  4 +++
 include/linux/kvm_host.h        |  1 +
 virt/kvm/kvm_main.c             | 14 ++++++++--
 7 files changed, 76 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -0
index 1ed1206c196db..6aca4f0f9d806 100644
@@ -91,6 +91,8 @@
 	KVM_ARCH_REQ_FLAGS(27, KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_APF_READY		KVM_ARCH_REQ(28)
 #define KVM_REQ_MSR_FILTER_CHANGED	KVM_ARCH_REQ(29)
+#define KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE \
+	KVM_ARCH_REQ_FLAGS(30, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 
 #define CR0_RESERVED_BITS                                               \
 	(~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
@@ -1029,6 +1031,7 @@ struct kvm_arch {
 	} msr_filter;
 
 	bool bus_lock_detection_enabled;
+	bool pml_enabled;
 
 	struct kvm_pmu_event_filter *pmu_event_filter;
 	struct task_struct *nx_lpage_recovery_thread;
@@ -1295,6 +1298,7 @@ struct kvm_x86_ops {
 					   struct kvm_memory_slot *slot,
 					   gfn_t offset, unsigned long mask);
 	int (*cpu_dirty_log_size)(void);
+	void (*update_vcpu_dirty_logging_state)(struct kvm_vcpu *vcpu);
 
 	/* pmu operations of sub-arch */
 	const struct kvm_pmu_ops *pmu_ops;
arch/x86/kvm/vmx/nested.c +5 -0
index b2f0b5e9cd638..9e8e89fdc03a2 100644
@@ -4495,6 +4495,11 @@ void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason,
 		vmx_set_virtual_apic_mode(vcpu);
 	}
 
+	if (vmx->nested.deferred_update_pml_vmcs) {
+		vmx->nested.deferred_update_pml_vmcs = false;
+		vmx_update_pml_in_vmcs(vcpu);
+	}
+
 	/* Unpin physical memory we referred to in vmcs02 */
 	if (vmx->nested.apic_access_page) {
 		kvm_release_page_clean(vmx->nested.apic_access_page);
arch/x86/kvm/vmx/vmx.c +47 -2

82 lines not on this page · raw message

arch/x86/kvm/vmx/vmx.h +3 -0
index 12c53d05a902b..b7dc413cda7bd 100644
@@ -181,6 +181,8 @@ struct nested_vmx {
 
 	struct loaded_vmcs vmcs02;
 
+	bool deferred_update_pml_vmcs;
+
 	/*
 	 * Guest pages referred to in the vmcs02 with host-physical
 	 * pointers, so we must keep them pinned while L2 runs.
@@ -393,6 +395,7 @@ int vmx_find_loadstore_msr_slot(struct vmx_msrs *m, u32 msr);
 void vmx_ept_load_pdptrs(struct kvm_vcpu *vcpu);
 void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu,
 	u32 msr, int type, bool value);
+void vmx_update_pml_in_vmcs(struct kvm_vcpu *vcpu);
 
 static inline u8 vmx_get_rvi(void)
 {
arch/x86/kvm/x86.c +4 -0
index d9f931c632936..75b924c9c5af0 100644
@@ -8984,6 +8984,10 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 			kvm_check_async_pf_completion(vcpu);
 		if (kvm_check_request(KVM_REQ_MSR_FILTER_CHANGED, vcpu))
 			static_call(kvm_x86_msr_filter_changed)(vcpu);
+
+		if (kvm_check_request(KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE,
+				      vcpu))
+			kvm_x86_ops.update_vcpu_dirty_logging_state(vcpu);
 	}
 
 	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
include/linux/kvm_host.h +1 -0
index f417447129b9c..a8a28ba955923 100644
@@ -518,6 +518,7 @@ struct kvm {
 	pid_t userspace_pid;
 	unsigned int max_halt_poll_ns;
 	u32 dirty_ring_size;
+	uint dirty_logging_enable_count;
 };
 
 #define kvm_err(fmt, ...) \
virt/kvm/kvm_main.c +12 -2
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
 {
 	return kvm_make_all_cpus_request_except(kvm, req, NULL);
 }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
 
 #ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
 void kvm_flush_remote_tlbs(struct kvm *kvm)
@@ -1366,15 +1367,24 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	}
 
 	/* Allocate/free page dirty bitmap as needed */
-	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES))
+	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES)) {
 		new.dirty_bitmap = NULL;
-	else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
+
+		if (old.flags & KVM_MEM_LOG_DIRTY_PAGES) {
+			WARN_ON(kvm->dirty_logging_enable_count == 0);
+			--kvm->dirty_logging_enable_count;
+		}
+
+	} else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
 		r = kvm_alloc_dirty_bitmap(&new);
 		if (r)
 			return r;
 
 		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
 			bitmap_set(new.dirty_bitmap, 0, new.npages);
+
+		++kvm->dirty_logging_enable_count;
+		WARN_ON(kvm->dirty_logging_enable_count == 0);
 	}
 
 	r = kvm_set_memslot(kvm, mem, &old, &new, as_id, change);
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
#1 lore raw
Quoted 19 lines · Makarand Sonare
arch/x86/kvm/vmx/vmx.c +13 -0
@@ -7517,9 +7531,39 @@ static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 static void vmx_slot_disable_log_dirty(struct kvm *kvm,
 				       struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and disable PML if dirty logging
+	 * is being disabled for the last slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 0 &&
+	    kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = false;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}
+
 	kvm_mmu_slot_set_dirty(kvm, slot);
 }

...

virt/kvm/kvm_main.c bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 310

Quoted 9 lines
virt/kvm/kvm_main.c +1 -0
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
 {
 	return kvm_make_all_cpus_request_except(kvm, req, NULL);
 }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);

If we move enable_pml into x86.c then this export and several of the kvm_x86_ops go away. I know this because I have a series I was about to send that does that, among several other things. I suspect that kvm->arch.pml_enabled could also go away, but that's just a guess.

Anyways, I'll work with you off-list to figure out a plan. The easiest thing is probably for me to tack it on to the end of my series. I completely spaced on the fact that my series would conflict with this code, sorry :-/

maintainer
#2 lore raw
Quoted 5 lines · Sean Christopherson
Quoted 5 lines
virt/kvm/kvm_main.c +0 -0
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
Quoted 8 lines

virt/kvm/kvm_main.c bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 310

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
  {
  	return kvm_make_all_cpus_request_except(kvm, req, NULL);
  }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);

If we move enable_pml into x86.c then this export and several of the kvm_x86_ops go away. I know this because I have a series I was about to send that does that, among several other things. I suspect that kvm->arch.pml_enabled could also go away, but that's just a guess.

I don't like the idea of moving enable_pml into x86.c, but I'm ready to be convinced otherwise. In any case, for sure you can _check_ enable_pml from x86.c via kvm_x86_ops.flush_log_dirty or kvm_x86_ops.cpu_dirty_log_size.

Paolo

#3 lore raw
Quoted 15 lines · Paolo Bonzini
Quoted 5 lines · Sean Christopherson
Quoted 5 lines
virt/kvm/kvm_main.c +0 -0
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
Quoted 8 lines

virt/kvm/kvm_main.c bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req) · line 310

Quoted 4 lines
virt/kvm/kvm_main.c +1 -0
  {
  	return kvm_make_all_cpus_request_except(kvm, req, NULL);
  }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);

If we move enable_pml into x86.c then this export and several of the kvm_x86_ops go away. I know this because I have a series I was about to send that does that, among several other things. I suspect that kvm->arch.pml_enabled could also go away, but that's just a guess.

Quoted 3 lines

I don't like the idea of moving enable_pml into x86.c, but I'm ready to be convinced otherwise. In any case, for sure you can _check_ enable_pml from x86.c via kvm_x86_ops.flush_log_dirty or kvm_x86_ops.cpu_dirty_log_size.

Ya, after taking another look at my series, exposing enable_pml isn't necessary. What I really dislike is bouncing through VMX and exporting MMU functions for no real benefit. The x86/MMU functions/behavior are tightly coupled to VMX's implementation, bouncing through kvm_x86_ops doesn't magically decouple things.

Anyways, kvm_x86_ops.cpu_dirty_log_size can be change to a simple integer instead of a callback function, and with that change I'm happy using cpu_dirty_log_size as the check with x86.c/mmu.c to determine whether or not hardware dirty logging is supported.

Thanks for the early sanity check :-)

#4 lore raw
In reply to Makarand Sonare
Quoted 5 lines · Makarand Sonare

Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Clearing the dirty bits is an expensive operation which can cause severe MMU

Quoted 8 lines

lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration. Also, this would break if some other code path clears the dirty bits in which case, PML will actually start logging dirty pages even when dirty logging is disabled incurring unnecessary vmexits when the PML buffer becomes full. In order to avoid this extra overhead, we should enable or disable PML in VMCS when dirty logging gets enabled or disabled instead of keeping it always enabled.

Breaking this up into a few paragraphs would be helpful.

Quoted 4 lines

Tested:
kvm-unit-tests
dirty_log_test
dirty_log_perf_test

Eh, I get that we like these for internal tracking, but for upstream there's an assumption that you did your due diligence. If there's something noteworthy about your testing (or lack thereof), throw it in the cover letter or in the part that's not recorded in the final commit.

Quoted 3 lines

---

...

Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +4 -0
@@ -7517,9 +7531,39 @@ static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 static void vmx_slot_disable_log_dirty(struct kvm *kvm,
 				       struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and disable PML if dirty logging
+	 * is being disabled for the last slot
+	 *

arch/x86/kvm/vmx/vmx.c static void vmx_slot_enable_log_dirty(struct kvm *kvm, · line 7547

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +9 -0
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 0 &&
+	    kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = false;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}
+
 	kvm_mmu_slot_set_dirty(kvm, slot);

The justification for dynamically toggling PML is that it means KVM can skip setting all the dirty bits when logging is disabled, but that code is still here. Is there a follow-up planned to reap the reward?

Quoted 1 line
(quoted diff) +0 -0
 }
maintainer
#5 lore raw
Quoted 9 lines · Sean Christopherson
Quoted 4 lines

Tested:
kvm-unit-tests
dirty_log_test
dirty_log_perf_test

Eh, I get that we like these for internal tracking, but for upstream there's an assumption that you did your due diligence. If there's something noteworthy about your testing (or lack thereof), throw it in the cover letter or in the part that's not recorded in the final commit.

I actually don't mind it and I should do it myself as well. Sure for large series it's better to put it just once in the cover letter, but for small submissions such as this one it's not a problem.

Paolo

maintainer
#6 lore raw
In reply to Makarand Sonare
Quoted 6 lines · Makarand Sonare
arch/x86/kvm/vmx/vmx.c +6 -0
+void vmx_update_pml_in_vmcs(struct kvm_vcpu *vcpu)
+{
+	if (cpu_has_secondary_exec_ctrls()) {
+		if (is_guest_mode(vcpu)) {
+			to_vmx(vcpu)->nested.deferred_update_pml_vmcs = true;
+			return;

arch/x86/kvm/vmx/vmx.c static void vmx_slot_enable_log_dirty(struct kvm *kvm, · line 7565

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+		}
+
+		if (vcpu->kvm->arch.pml_enabled)
+			vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL,
+				SECONDARY_EXEC_ENABLE_PML);
+		else
+			vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL,
+				SECONDARY_EXEC_ENABLE_PML);
+	}
+}

Since the execution controls are shadowed, they can be read quite efficiently. This means that there's no need for vcpu->kvm->arch.pml_enabled, and also that the copy can be done unconditionally in prepare_vmcs02 and nested_vmx_vmexit.

If the above is not true, we should at least combine change_vmcs01_virtual_apic_mode, reload_vmcs01_apic_access_page and the new field in a single bit field, for example vmx->nested.dirty_vmcs01_fields or vmx->nested.vmexit_requests.

In any case I expect Sean to take care of submitting this patch and I have to do nothing more about it, right?

Paolo

#7 lore raw
Quoted 24 lines · Paolo Bonzini
Quoted 6 lines · Makarand Sonare
arch/x86/kvm/vmx/vmx.c +6 -0
+void vmx_update_pml_in_vmcs(struct kvm_vcpu *vcpu)
+{
+	if (cpu_has_secondary_exec_ctrls()) {
+		if (is_guest_mode(vcpu)) {
+			to_vmx(vcpu)->nested.deferred_update_pml_vmcs = true;
+			return;

arch/x86/kvm/vmx/vmx.c static void vmx_slot_enable_log_dirty(struct kvm *kvm, · line 7565

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+		}
+
+		if (vcpu->kvm->arch.pml_enabled)
+			vmcs_set_bits(SECONDARY_VM_EXEC_CONTROL,
+				SECONDARY_EXEC_ENABLE_PML);
+		else
+			vmcs_clear_bits(SECONDARY_VM_EXEC_CONTROL,
+				SECONDARY_EXEC_ENABLE_PML);
+	}
+}

Since the execution controls are shadowed, they can be read quite efficiently. This means that there's no need for vcpu->kvm->arch.pml_enabled, and also that the copy can be done unconditionally in prepare_vmcs02 and nested_vmx_vmexit.

Quoted 7 lines

If the above is not true, we should at least combine change_vmcs01_virtual_apic_mode, reload_vmcs01_apic_access_page and the new field in a single bit field, for example vmx->nested.dirty_vmcs01_fields or vmx->nested.vmexit_requests.

In any case I expect Sean to take care of submitting this patch and I have to do nothing more about it, right?

Right, we'll sort it out.

#8 lore raw
In reply to Makarand Sonare
Quoted 4 lines · Makarand Sonare

Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Clearing

s/clearing/setting

Clearing is also expensive, but that can't be optimized away with this change.

Quoted 9 lines

the dirty bits is an expensive operation which can cause severe MMU lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration. Also, this would break if some other code path clears the dirty bits in which case, PML will actually start logging dirty pages even when dirty logging is disabled incurring unnecessary vmexits when the PML buffer becomes full. In order to avoid this extra overhead, we should enable or disable PML in VMCS when dirty logging gets enabled or disabled instead of keeping it always enabled.

...

arch/x86/kvm/vmx/vmx.c static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx) · line 4280

Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +1 -1
index 777177ea9a35e..eb6639f0ee7eb 100644
@@ -4276,7 +4276,7 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)
 	*/
 	exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS;
 
-	if (!enable_pml)
+	if (!enable_pml || !vcpu->kvm->arch.pml_enabled)
 		exec_control &= ~SECONDARY_EXEC_ENABLE_PML;

The checks are unnecessary if PML is dynamically toggled, i.e. this snippet can unconditionally clear PML. When setting SECONDARY_EXEC (below snippet), PML will be preserved in the current controls, which is what we want.

Quoted 17 lines
arch/x86/kvm/vmx/vmx.c +4 -1
 	if (cpu_has_vmx_xsaves()) {
@@ -7133,7 +7133,8 @@ static void vmcs_set_secondary_exec_control(struct vcpu_vmx *vmx)
 		SECONDARY_EXEC_SHADOW_VMCS |
 		SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE |
 		SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
-		SECONDARY_EXEC_DESC;
+		SECONDARY_EXEC_DESC |
+		SECONDARY_EXEC_ENABLE_PML;
 
 	u32 new_ctl = vmx->secondary_exec_control;
 	u32 cur_ctl = secondary_exec_controls_get(vmx);
@@ -7509,6 +7510,19 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu)
 static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 				     struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and enable PML if dirty logging

arch/x86/kvm/vmx/vmx.c static void vmx_sched_in(struct kvm_vcpu *vcpu, int cpu) · line 7524

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+	 * is being enabled for the 1st slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 1 &&
+	    !kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = true;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}

This is flawed. .slot_enable_log_dirty() and .slot_disable_log_dirty() are only called when LOG_DIRTY_PAGE is toggled in an existing memslot _and_ only the flags of the memslot are being changed. This fails to enable PML if the first memslot with LOG_DIRTY_PAGE is created or moved, and fails to disable PML if the last memslot with LOG_DIRTY_PAGE is deleted.

Quoted 23 lines
arch/x86/kvm/vmx/vmx.c +14 -0
+
 	if (!kvm_dirty_log_manual_protect_and_init_set(kvm))
 		kvm_mmu_slot_leaf_clear_dirty(kvm, slot);
 	kvm_mmu_slot_largepage_remove_write_access(kvm, slot);
@@ -7517,9 +7531,39 @@ static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 static void vmx_slot_disable_log_dirty(struct kvm *kvm,
 				       struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and disable PML if dirty logging
+	 * is being disabled for the last slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 0 &&
+	    kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = false;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}
+
 	kvm_mmu_slot_set_dirty(kvm, slot);
 }

...

Quoted 15 lines

#define kvm_err(fmt, ...) \

virt/kvm/kvm_main.c +1 -0
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)
 {
 	return kvm_make_all_cpus_request_except(kvm, req, NULL);
 }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
 
 #ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
 void kvm_flush_remote_tlbs(struct kvm *kvm)
@@ -1366,15 +1367,24 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	}

virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 1375

Quoted 10 lines
virt/kvm/kvm_main.c +5 -2
 
 	/* Allocate/free page dirty bitmap as needed */
-	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES))
+	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES)) {
 		new.dirty_bitmap = NULL;
-	else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
+
+		if (old.flags & KVM_MEM_LOG_DIRTY_PAGES) {
+			WARN_ON(kvm->dirty_logging_enable_count == 0);
+			--kvm->dirty_logging_enable_count;

The count will be corrupted if kvm_set_memslot() fails.

The easiest/cleanest way to fix both this and the refcounting bug is to handle the count in kvm_mmu_slot_apply_flags(). That will also allow making the dirty log count x86-only, and it can then be renamed to cpu_dirty_log_count to align with the

We can always move/rename the count variable if additional motivation for tracking dirty logging comes along.

Quoted 17 lines
virt/kvm/kvm_main.c +6 -0
+		}
+
+	} else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
 		r = kvm_alloc_dirty_bitmap(&new);
 		if (r)
 			return r;
 
 		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
 			bitmap_set(new.dirty_bitmap, 0, new.npages);
+
+		++kvm->dirty_logging_enable_count;
+		WARN_ON(kvm->dirty_logging_enable_count == 0);
 	}
 
 	r = kvm_set_memslot(kvm, mem, &old, &new, as_id, change);
Signature · 1 line
2.30.0.478.g8a0d178c01-goog
#9 lore raw
Quoted 9 lines
Quoted 4 lines

Currently, if enable_pml=1 PML remains enabled for the entire lifetime of the VM irrespective of whether dirty logging is enable or disabled. When dirty logging is disabled, all the pages of the VM are manually marked dirty, so that PML is effectively non-operational. Clearing

s/clearing/setting

Clearing is also expensive, but that can't be optimized away with this change.

Thanks for catching the typo, it should be setting.

Quoted 23 lines
Quoted 10 lines

the dirty bits is an expensive operation which can cause severe MMU lock contention in a performance sensitive path when dirty logging is disabled after a failed or canceled live migration. Also, this would break if some other code path clears the dirty bits in which case, PML will actually start logging dirty pages even when dirty logging is disabled incurring unnecessary vmexits when the PML buffer becomes full. In order to avoid this extra overhead, we should enable or disable PML in VMCS when dirty logging gets enabled or disabled instead of keeping it always enabled.

...

Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +0 -0
index 777177ea9a35e..eb6639f0ee7eb 100644
@@ -4276,7 +4276,7 @@ static void
 vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)
 	*/
 	exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS;
 
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx) · line 4280

Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +1 -1
-	if (!enable_pml)
+	if (!enable_pml || !vcpu->kvm->arch.pml_enabled)
 		exec_control &= ~SECONDARY_EXEC_ENABLE_PML;

The checks are unnecessary if PML is dynamically toggled, i.e. this snippet can
unconditionally clear PML. When setting SECONDARY_EXEC (below snippet), PML
will be preserved in the current controls, which is what we want.

Assuming a new VCPU can be added at a later time after PML is already enabled, should we clear
PML in VMCS for the new VCPU. If yes what will be the trigger for setting PML for the new VCPU?

Quoted 31 lines
Quoted 20 lines
arch/x86/kvm/vmx/vmx.c +2 -1
 	if (cpu_has_vmx_xsaves()) {
@@ -7133,7 +7133,8 @@ static void vmcs_set_secondary_exec_control(struct
 vcpu_vmx *vmx)
 		SECONDARY_EXEC_SHADOW_VMCS |
 		SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE |
 		SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
-		SECONDARY_EXEC_DESC;
+		SECONDARY_EXEC_DESC |
+		SECONDARY_EXEC_ENABLE_PML;
 
 	u32 new_ctl = vmx->secondary_exec_control;
arch/x86/kvm/vmx/vmx.c +2 -0
 	u32 cur_ctl = secondary_exec_controls_get(vmx);
@@ -7509,6 +7510,19 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int
 cpu)
 static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 				     struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and enable PML if dirty logging

arch/x86/kvm/vmx/vmx.c static void vmx_sched_in(struct kvm_vcpu *vcpu, int · line 7525

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+	 * is being enabled for the 1st slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 1 &&
+	    !kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = true;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}
Quoted 8 lines

This is flawed. .slot_enable_log_dirty() and .slot_disable_log_dirty() are only
called when LOG_DIRTY_PAGE is toggled in an existing memslot _and_ only the flags of the memslot are being changed. This fails to enable PML if the first
memslot with LOG_DIRTY_PAGE is created or moved, and fails to disable PML if the
last memslot with LOG_DIRTY_PAGE is deleted.

Thanks for pointing out. If there is such a scenario, what do you suggest to handle this?

Quoted 65 lines
Quoted 25 lines
arch/x86/kvm/vmx/vmx.c +1 -0
+
 	if (!kvm_dirty_log_manual_protect_and_init_set(kvm))
 		kvm_mmu_slot_leaf_clear_dirty(kvm, slot);
arch/x86/kvm/vmx/vmx.c +13 -0
 	kvm_mmu_slot_largepage_remove_write_access(kvm, slot);
@@ -7517,9 +7531,39 @@ static void vmx_slot_enable_log_dirty(struct kvm
 *kvm,
 static void vmx_slot_disable_log_dirty(struct kvm *kvm,
 				       struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and disable PML if dirty logging
+	 * is being disabled for the last slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 0 &&
+	    kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = false;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}
+
 	kvm_mmu_slot_set_dirty(kvm, slot);
 }

...

Quoted 16 lines

#define kvm_err(fmt, ...) \

virt/kvm/kvm_main.c +1 -0
index ee4ac2618ec59..c6e5b026bbfe8 100644
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kvm *kvm,
 unsigned int req)
 {
 	return kvm_make_all_cpus_request_except(kvm, req, NULL);
 }
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
 
 #ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL

void kvm_flush_remote_tlbs(struct kvm *kvm)

virt/kvm/kvm_main.c +0 -0
@@ -1366,15 +1367,24 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	}

virt/kvm/kvm_main.c int __kvm_set_memory_region(struct kvm *kvm, · line 1375

Quoted 10 lines
virt/kvm/kvm_main.c +5 -2
 
 	/* Allocate/free page dirty bitmap as needed */
-	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES))
+	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES)) {
 		new.dirty_bitmap = NULL;
-	else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
+
+		if (old.flags & KVM_MEM_LOG_DIRTY_PAGES) {
+			WARN_ON(kvm->dirty_logging_enable_count == 0);
+			--kvm->dirty_logging_enable_count;

The count will be corrupted if kvm_set_memslot() fails.

The easiest/cleanest way to fix both this and the refcounting bug is to handle
the count in kvm_mmu_slot_apply_flags(). That will also allow making the dirty
log count x86-only, and it can then be renamed to cpu_dirty_log_count to align
with the

Quoted 2 lines

We can always move/rename the count variable if additional motivation for tracking dirty logging comes along.

Thanks for pointing out. Will this solution take care of the scenario where a memslot is created/deleted with LOG_DIRTY_PAGE?

Quoted 20 lines
Quoted 19 lines
virt/kvm/kvm_main.c +6 -1
+		}
+
+	} else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
 		r = kvm_alloc_dirty_bitmap(&new);
 		if (r)
 			return r;
 
 		if (kvm_dirty_log_manual_protect_and_init_set(kvm))
 			bitmap_set(new.dirty_bitmap, 0, new.npages);
+
+		++kvm->dirty_logging_enable_count;
+		WARN_ON(kvm->dirty_logging_enable_count == 0);
 	}
 
 	r = kvm_set_memslot(kvm, mem, &old, &new, as_id, change);
--

2.30.0.478.g8a0d178c01-goog

#10 lore raw
Quoted 19 lines · Makarand Sonare
Quoted 9 lines
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c +0 -0
index 777177ea9a35e..eb6639f0ee7eb 100644
@@ -4276,7 +4276,7 @@ static void
 vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)
 	*/
 	exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS;
 
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx) · line 4280

Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +1 -1
-	if (!enable_pml)
+	if (!enable_pml || !vcpu->kvm->arch.pml_enabled)
 		exec_control &= ~SECONDARY_EXEC_ENABLE_PML;

The checks are unnecessary if PML is dynamically toggled, i.e. this snippet can
unconditionally clear PML. When setting SECONDARY_EXEC (below snippet), PML
will be preserved in the current controls, which is what we want.

Quoted 4 lines

Assuming a new VCPU can be added at a later time after PML is already enabled, should we clear
PML in VMCS for the new VCPU. If yes what will be the trigger for setting PML for the new VCPU?

Ah, didn't consider that. Phooey.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
arch/x86/kvm/vmx/vmx.c +2 -1
 	if (cpu_has_vmx_xsaves()) {
@@ -7133,7 +7133,8 @@ static void vmcs_set_secondary_exec_control(struct
 vcpu_vmx *vmx)
 		SECONDARY_EXEC_SHADOW_VMCS |
 		SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE |
 		SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES |
-		SECONDARY_EXEC_DESC;
+		SECONDARY_EXEC_DESC |
+		SECONDARY_EXEC_ENABLE_PML;
 
 	u32 new_ctl = vmx->secondary_exec_control;
arch/x86/kvm/vmx/vmx.c +12 -0
 	u32 cur_ctl = secondary_exec_controls_get(vmx);
@@ -7509,6 +7510,19 @@ static void vmx_sched_in(struct kvm_vcpu *vcpu, int
 cpu)
 static void vmx_slot_enable_log_dirty(struct kvm *kvm,
 				     struct kvm_memory_slot *slot)
 {
+	/*
+	 * Check all slots and enable PML if dirty logging
+	 * is being enabled for the 1st slot
+	 *
+	 */
+	if (enable_pml &&
+	    kvm->dirty_logging_enable_count == 1 &&
+	    !kvm->arch.pml_enabled) {
+		kvm->arch.pml_enabled = true;
+		kvm_make_all_cpus_request(kvm,
+			KVM_REQ_UPDATE_VCPU_DIRTY_LOGGING_STATE);
+	}

...

Quoted 20 lines
Quoted 20 lines

arch/x86/kvm/vmx/vmx.c int __kvm_set_memory_region(struct kvm *kvm, · line 1375

Quoted 12 lines
arch/x86/kvm/vmx/vmx.c +5 -2
@@ -1366,15 +1367,24 @@ int __kvm_set_memory_region(struct kvm *kvm,
 	}
 
 	/* Allocate/free page dirty bitmap as needed */
-	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES))
+	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES)) {
 		new.dirty_bitmap = NULL;
-	else if (!new.dirty_bitmap && !kvm->dirty_ring_size) {
+
+		if (old.flags & KVM_MEM_LOG_DIRTY_PAGES) {
+			WARN_ON(kvm->dirty_logging_enable_count == 0);
+			--kvm->dirty_logging_enable_count;

The count will be corrupted if kvm_set_memslot() fails.

The easiest/cleanest way to fix both this and the refcounting bug is to handle the count in kvm_mmu_slot_apply_flags(). That will also allow making the dirty log count x86-only, and it can then be renamed to cpu_dirty_log_count to align with the

Quoted 5 lines
Quoted 2 lines

We can always move/rename the count variable if additional motivation for tracking dirty logging comes along.

Thanks for pointing out. Will this solution take care of the scenario where a memslot is created/deleted with LOG_DIRTY_PAGE?

Yes? At least, that's the plan. :-) I'll post my whole series as an RFC later today so you and Ben can poke holes in my changes. There are some TDP MMU fixes that I've accumulated and would like to get posted before the 5.12 merge window opens, if only so that Paolo can make an informed decision on whether or not to enable TDP MMU by default.

#11 lore raw
Quoted 19 lines · Sean Christopherson
Quoted 18 lines · Makarand Sonare
Quoted 17 lines

arch/x86/kvm/vmx/vmx.c static void · line 4281

Quoted 12 lines
arch/x86/kvm/vmx/vmx.c +1 -1
index 777177ea9a35e..eb6639f0ee7eb 100644
@@ -4276,7 +4276,7 @@ static void
 vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)
 	*/
 	exec_control &= ~SECONDARY_EXEC_SHADOW_VMCS;
 
-	if (!enable_pml)
+	if (!enable_pml || !vcpu->kvm->arch.pml_enabled)
 		exec_control &= ~SECONDARY_EXEC_ENABLE_PML;

The checks are unnecessary if PML is dynamically toggled, i.e. this snippet can unconditionally clear PML. When setting SECONDARY_EXEC (below snippet), PML will be preserved in the current controls, which is what we want.

Quoted 6 lines
Quoted 4 lines

Assuming a new VCPU can be added at a later time after PML is already enabled, should we clear
PML in VMCS for the new VCPU. If yes what will be the trigger for setting PML for the new VCPU?

Ah, didn't consider that. Phooey.

I remember why I thought this could be unconditional. Adding PML to the list of dynamic bits in vmcs_set_secondary_exec_control() effectively makes this code unconditional, because it means that current bit will be preserved, including the case where PML=0 when a vCPU is created.

I believe the fix is simply to not mark PML as fully dynamic.