Skip to contentSkip to conversation

KVM: X86: set EXITING_GUEST_MODE as soon as vCPU exits

PATCH In review Jon Kohler 5 participants 12 messages Nov 29, 2022 – Dec 9, 2022

Subsystems (3)
lore
Share & tools
lore

Jon Kohler

author
#0 lore raw

PATCH 1/1 KVM: X86: set EXITING_GUEST_MODE as soon as vCPU exits

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Changing vcpu->mode away from IN_GUEST_MODE as early as possible gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

---
 arch/x86/kvm/svm/svm.c |  7 +++++++
 arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
 arch/x86/kvm/x86.c     |  8 ++++++++
 3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +7 -0
index ce362e88a567..5f0c118a3ffd 100644
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +23 -0
index 63247c57c72c..243dcb87c727 100644
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;
 		kvm_lapic_expired_hv_timer(vcpu);
 		return EXIT_FASTPATH_REENTER_GUEST;
 	}
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +8 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;
 			ret = EXIT_FASTPATH_REENTER_GUEST;
 		}
 		break;
Signature · 1 line
2.30.1 (Apple Git-130)
#1 lore raw
Quoted 10 lines · Jon Kohler

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Quoted 7 lines

Changing vcpu->mode away from IN_GUEST_MODE as early as possible gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

Beware that we had a king sized bug in regard to AVIC inhibition races vs guest entries, this this should be carefully checked for this.

Also, do you have any perf numbers to see if that actually improves performance? (I am just curious, I do think that this can improve performance).

Best regards,
Maxim Levitsky

Quoted 86 lines
---
 arch/x86/kvm/svm/svm.c |  7 +++++++
 arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
 arch/x86/kvm/x86.c     |  8 ++++++++
 3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +7 -0
index ce362e88a567..5f0c118a3ffd 100644
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +23 -0
index 63247c57c72c..243dcb87c727 100644
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;
 		kvm_lapic_expired_hv_timer(vcpu);
 		return EXIT_FASTPATH_REENTER_GUEST;
 	}
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +8 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;
 			ret = EXIT_FASTPATH_REENTER_GUEST;
 		}
 		break;
author
#2 lore raw
Quoted 23 lines · Maxim Levitsky
Quoted 10 lines · Jon Kohler

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Quoted 7 lines

Changing vcpu->mode away from IN_GUEST_MODE as early as possible gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

Quoted 2 lines

Beware that we had a king sized bug in regard to AVIC inhibition races vs guest entries, this this should be carefully checked for this.

Thanks, Maxim - any pointers on what we should be looking for here?

Quoted 4 lines

Also, do you have any perf numbers to see if that actually improves performance? (I am just curious, I do think that this can improve performance).

Yes indeed! Sorry I should have put that right in the commit msg as a note, but using the kvm-unit-tests vmexit_ipi with -smp 20 on an Intel 8168 its roughly ~3% better (~3325-ish vs ~3400-ish), though the test is a bit noisy even with taskset to a single socket.

To help validate that we were even getting *any* benefit, in a local build I added a log statement (rough code below) to IPI delivery path, and did see many, many IPIs getting suppressed that would have otherwise fired.

kvm_vcpu_trigger_posted_interrupt() {
...

    if (vcpu->mode == EXITING_GUEST_MODE) {
        pr_warn_ratelimited("exiting suppression worked");
    }

...
}

Quoted 90 lines

Best regards,
Maxim Levitsky

Quoted 86 lines

---

arch/x86/kvm/svm/svm.c |  7 +++++++
arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
arch/x86/kvm/x86.c     |  8 ++++++++
3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +7 -0
index ce362e88a567..5f0c118a3ffd 100644
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +23 -0
index 63247c57c72c..243dcb87c727 100644
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;
 		kvm_lapic_expired_hv_timer(vcpu);
 		return EXIT_FASTPATH_REENTER_GUEST;
 	}
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +8 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;
 			ret = EXIT_FASTPATH_REENTER_GUEST;
 		}
 		break;
#3 lore raw
Quoted 21 lines · Jon Kohler
Quoted 21 lines · Maxim Levitsky
Quoted 17 lines · Jon Kohler

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Changing vcpu->mode away from IN_GUEST_MODE as early as possible gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

Quoted 4 lines
Quoted 2 lines

Beware that we had a king sized bug in regard to AVIC inhibition races vs guest entries, this this should be carefully checked for this.

Thanks, Maxim - any pointers on what we should be looking for here?

I need to swap the whole thing in to be able to comment on this one. I'll do this next week.

Overall the problem is that the target vCPU can inhibit its AVIC at any moment, which forces the senders to use normal KVM_REQ_EVENT + vcpu kick to deliver it the interrupt. It is very racy.

The only other pointer I can recall now is the code at 'svm_complete_interrupt_delivery'

Best regards,
Maxim Levitsky

Quoted 111 lines
Quoted 4 lines

Also, do you have any perf numbers to see if that actually improves performance? (I am just curious, I do think that this can improve performance).

Yes indeed! Sorry I should have put that right in the commit msg as a note, but using the kvm-unit-tests vmexit_ipi with -smp 20 on an Intel 8168 its roughly ~3% better (~3325-ish vs ~3400-ish), though the test is a bit noisy even with taskset to a single socket.

To help validate that we were even getting *any* benefit, in a local build I added a log statement (rough code below) to IPI delivery path, and did see many, many IPIs getting suppressed that would have otherwise fired.

kvm_vcpu_trigger_posted_interrupt() {
...

    if (vcpu->mode == EXITING_GUEST_MODE) {
        pr_warn_ratelimited("exiting suppression worked");
    }

...
}

Quoted 89 lines

Best regards,
Maxim Levitsky

Quoted 85 lines

---

arch/x86/kvm/svm/svm.c |  7 +++++++
arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
arch/x86/kvm/x86.c     |  8 ++++++++
3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +7 -0
index ce362e88a567..5f0c118a3ffd 100644
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +23 -0
index 63247c57c72c..243dcb87c727 100644
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;
 		kvm_lapic_expired_hv_timer(vcpu);
 		return EXIT_FASTPATH_REENTER_GUEST;
 	}
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +8 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;
 			ret = EXIT_FASTPATH_REENTER_GUEST;
 		}
 		break;
#4 lore raw
In reply to Jon Kohler
Quoted 5 lines · Jon Kohler
arch/x86/kvm/vmx/vmx.c +1 -0
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;

arch/x86/kvm/vmx/vmx.c void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) · line 7055

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10 -0
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;

Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit guest mode. But ...

Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +1 -0
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +0 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);

arch/x86/kvm/x86.c fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu) · line 2170

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;

... the vCPU enters guest mode again.

I believe mode transition from EXITING_GUEST_MODE to IN_GUEST_MODE directly isn't valid for current KVM.

author
#5 lore raw
Quoted 4 lines · Chao Gao

Chao while I’ve got you here, I was inspired to tune up the software side here based on the VTD suppress notifications change we had been talking about. Any chance we could get the v4 of that? Seemed like it was almost done, yea? Would love to get our hands on that to help accelerate the VTD path.

Quoted 12 lines · Jon Kohler
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +7 -0
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
Quoted 8 lines

arch/x86/kvm/vmx/vmx.c void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) · line 7055

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4 -0
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;

Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit guest mode. But ...

IIRC that’d only be a problem for fast path exits that reenter guest (like TSC Deadline) everything else *will* eventually exit out to kernel mode to pickup whatever other requests may be pending. In this sense, this patch is actually even better for kick because we will send incrementally less spurious kicks.

Even then, for fast path reentry exits, a guest is likely to exit all the way out eventually for something else soon enough, so worst case something gets a wee bit more delayed than it should. Small price to pay for clawing back cycles on the IPI send side I think.

Quoted 24 lines
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +1 -0
+
 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
 
 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
arch/x86/kvm/x86.c +0 -0
index 2835bd796639..0e0d228f3fa5 100644
@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
 		data = kvm_read_edx_eax(vcpu);

arch/x86/kvm/x86.c fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu) · line 2170

Quoted 10 lines
arch/x86/kvm/x86.c +8 -0
 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
 			kvm_skip_emulated_instruction(vcpu);
+			/* Reset IN_GUEST_MODE since we're going to reenter
+			 * guest as part of this fast path. This is done as
+			 * an optimization without a memory barrier since
+			 * EXITING_GUEST_MODE is also set without a memory
+			 * barrier.
+			 */
+			if (vcpu->mode == EXITING_GUEST_MODE)
+				vcpu->mode = IN_GUEST_MODE;

... the vCPU enters guest mode again.

Quoted 2 lines

I believe mode transition from EXITING_GUEST_MODE to IN_GUEST_MODE directly isn't valid for current KVM.

You’re correct that we do not have this pattern in KVM today; however, I couldn’t Find anything in the vcpu-requests documentation that specifically says this isn’t Allowed. Furthermore, this really is the spirit of what we’re doing, e.g.:

We are indeed existing guest mode, but because there is a fast path, we aren’t exiting anymore, so we need to flip the state back to in guest mode.

This code just makes that sentiment clear in code, and just so happens to also further suppress IPI sends during that small window.

#6 lore raw
Quoted 8 lines · Jon Kohler
Quoted 4 lines · Chao Gao

Chao while I’ve got you here, I was inspired to tune up the software side here based on the VTD suppress notifications change we had been talking about. Any chance we could get the v4 of that? Seemed like it was almost done, yea? Would love to

I didn't post a new version because there is no feedback on v3. But considering there is a mistake in v3, I will fix it and post v4.

Quoted 24 lines

get our hands on that to help accelerate the VTD path.

Quoted 12 lines · Jon Kohler
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +7 -0
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
Quoted 8 lines

arch/x86/kvm/vmx/vmx.c void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) · line 7055

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4 -0
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;

Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit guest mode. But ...

Quoted 4 lines

IIRC that’d only be a problem for fast path exits that reenter guest (like TSC Deadline) everything else *will* eventually exit out to kernel mode to pickup whatever other requests may be pending. In this sense, this patch is actually even better for kick because we will send incrementally less spurious kicks.

Yes. I agree.

Quoted 4 lines

Even then, for fast path reentry exits, a guest is likely to exit all the way out eventually for something else soon enough, so worst case something gets a wee bit more delayed than it should. Small price to pay for clawing back cycles on the IPI send side I think.

Thanks for above clarification. On second thoughts, for fastpath, there is a call of kvm_vcpu_exit_request() before re-entry. This call guarantees that vCPUs will exit guest mode if any request pending. So, this change actually won't lead to a delay in handling pending events.

author
#7 lore raw
Quoted 10 lines · Chao Gao
Quoted 5 lines · Jon Kohler
Quoted 4 lines · Chao Gao
Quoted 6 lines
Quoted 3 lines

Chao while I’ve got you here, I was inspired to tune up the software side here based on the VTD suppress notifications change we had been talking about. Any chance we could get the v4 of that? Seemed like it was almost done, yea? Would love to

I didn't post a new version because there is no feedback on v3. But considering there is a mistake in v3, I will fix it and post v4.

Ok Thanks! Looking forward to that. In between that patch and this one, should be a great combined impact. Any chance you can apply my patch and yours together and see how it works? I’d imagine it isn’t as applicable with IPI-v, but it’d still be interesting to see how the test numbers work out with your benchmark with/without IPI-v to see if your test sees a speedup here too.

Quoted 33 lines
Quoted 25 lines

get our hands on that to help accelerate the VTD path.

Quoted 12 lines · Jon Kohler
Quoted 11 lines
arch/x86/kvm/vmx/vmx.c +7 -0
@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
 void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
 					unsigned int flags)
 {
+	struct kvm_vcpu *vcpu = &vmx->vcpu;
+
+	/* Optimize IPI reduction by setting mode immediately after vmexit
+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 * Since the rdmsr and wrmsr below are expensive, this must be done
Quoted 8 lines

arch/x86/kvm/vmx/vmx.c void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) · line 7055

Quoted 4 lines
arch/x86/kvm/vmx/vmx.c +4 -0
+	 * first, so that the IPI suppression window covers the time dealing
+	 * with fixing up SPEC_CTRL.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;

Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit guest mode. But ...

Quoted 4 lines

IIRC that’d only be a problem for fast path exits that reenter guest (like TSC Deadline) everything else *will* eventually exit out to kernel mode to pickup whatever other requests may be pending. In this sense, this patch is actually even better for kick because we will send incrementally less spurious kicks.

Yes. I agree.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Even then, for fast path reentry exits, a guest is likely to exit all the way out eventually for something else soon enough, so worst case something gets a wee bit more delayed than it should. Small price to pay for clawing back cycles on the IPI send side I think.

Thanks for above clarification. On second thoughts, for fastpath, there is a call of kvm_vcpu_exit_request() before re-entry. This call guarantees that vCPUs will exit guest mode if any request pending. So, this change actually won't lead to a delay in handling pending events.

Ok thanks. I know this week tends to be a slow(er) week in the US coming back from the Holidays, so will wait for additional review / comments here

#8 lore raw
In reply to Jon Kohler
Quoted 5 lines · Jon Kohler

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

This is not remotely sufficient justification. Memory "barriers" are just compiler barriers on x86, so the odds of regressing the VM-Enter/VM-Exit cost without introducing a bug are miniscule.

Quoted 5 lines

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Changing vcpu->mode away from IN_GUEST_MODE as early as possible

Except this isn't as early as possible. If we're going to bother doing something like this, my vote is to move it into assembly.

Quoted 6 lines

gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

No, worst case scenario is that kvm_vcpu_exiting_guest_mode() sees EXITING_GUEST_MODE and doesn't kick the vCPU. For "kicks" that set a request, kvm_vcpu_exit_request() will punt the vCPU out of the tight run loop, though there might be ordering issues.

But whether or not there are ordering issues is a moot point since there are uses of kvm_vcpu_kick() that aren't accompanied by a request, e.g. to purge the PML buffers. In other words, kvm_vcpu_kick() absolutely cannot have false negatives. We could modify KVM to require a request when using kvm_vcpu_kick(), but that's a bit of a hack, and all of the possible ordering problems is still a pile of complexity I'd rather avoid.

No small part of me thinks we'd be better off adding a dedicated flag to very precisely track whether or not the vCPU is truly "in the guest" for the purposes of sending IPIs. Things like kicks have different requirements around IN_GUEST_MODE than sending interrupts, e.g. KVM manually processes the IRR on every VM-Enter and so lack of an IPI is a non-issue, whereas missing an IPI for a kick is problematic. In other words, EXITING_GUEST_MODE really needs to mean "existing the run loop".

E.g. toggle the dedicated flag within a few instructions of VM-Enter and VM-Exit for maximum efficiency for interrupts, and avoid having to make vcpu->mode more complex than it already is.

To add clarity, we could even rename IN_GUEST_MODE and EXITING_GUEST_MODE to something like IN_RUN_LOOP and EXITING_RUN_LOOP.

Quoted 12 lines
---
 arch/x86/kvm/svm/svm.c |  7 +++++++
 arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
 arch/x86/kvm/x86.c     |  8 ++++++++
 3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +0 -0
index ce362e88a567..5f0c118a3ffd 100644

arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in · line 3910

Quoted 4 lines
arch/x86/kvm/svm/svm.c +1 -0
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
+	/* Optimize IPI reduction by setting mode immediately after vmexit
	/*
 	 * Because KVM isn't the crazy land of net/ block comments should like
	 * this. 
	 */
Quoted 14 lines
arch/x86/kvm/svm/svm.c +6 -0
+	 * without a memmory barrier as this as not paired anywhere.
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +0 -0
index 63247c57c72c..243dcb87c727 100644

arch/x86/kvm/vmx/vmx.c static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu) · line 5884

Quoted 7 lines
arch/x86/kvm/vmx/vmx.c +4 -0
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory

Heh, justifying the lack of a memory barrier by saying pointing out the other code you added doesn't use a memory barrier is interesting, to put it politely.

arch/x86/kvm/vmx/vmx.c static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu) · line 5891

Quoted 7 lines
arch/x86/kvm/vmx/vmx.c +7 -0
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;

It's far easier, likely more performant, documents why this has a chance of working, and significantly less error prone to do this unconditionally in either assembly or after the EXIT_FASTPATH_REENTER_GUEST check in vcpu_enter_guest().

author
#9 lore raw
Quoted 11 lines · Sean Christopherson
Quoted 5 lines · Jon Kohler

Set vcpu->mode to EXITING_GUEST_MODE as soon vCPU exits to reflect that we are indeed exiting guest mode, but not quite out of guest mode yet. Note: This is done lazily without an explicit memory barrier so that we do not regress the cost in the critical path of going from the exit to the exit handler.

Quoted 3 lines

This is not remotely sufficient justification. Memory "barriers" are just compiler barriers on x86, so the odds of regressing the VM-Enter/VM-Exit cost without introducing a bug are miniscule.

I was talking about hardware memory barriers, not compiler memory barriers. I’ll tune that up and take a different approach on v2.

Quoted 9 lines
Quoted 6 lines

Flip back to IN_GUEST_MODE for exits that use EXIT_FASTPATH_REENTER_GUEST, such that we are IN_GUEST_MODE upon reentry.

Changing vcpu->mode away from IN_GUEST_MODE as early as possible

Except this isn't as early as possible. If we're going to bother doing something like this, my vote is to move it into assembly.

In vmenter.S, tacking on to call vmx_spec_ctrl_restore_host seemed like the most logical place after handling all of the state saves and RSB work. Are you saying put it even closer to the exit, meaning before the FILL_RETURN_BUFFER?

Quoted 19 lines
Quoted 7 lines

gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

No, worst case scenario is that kvm_vcpu_exiting_guest_mode() sees EXITING_GUEST_MODE and doesn't kick the vCPU. For "kicks" that set a request, kvm_vcpu_exit_request() will punt the vCPU out of the tight run loop, though there might be ordering issues.

But whether or not there are ordering issues is a moot point since there are uses of kvm_vcpu_kick() that aren't accompanied by a request, e.g. to purge the PML buffers. In other words, kvm_vcpu_kick() absolutely cannot have false negatives. We could modify KVM to require a request when using kvm_vcpu_kick(), but that's a bit of a hack, and all of the possible ordering problems is still a pile of complexity I'd rather avoid.

Quoted 6 lines

No small part of me thinks we'd be better off adding a dedicated flag to very precisely track whether or not the vCPU is truly "in the guest" for the purposes of sending IPIs. Things like kicks have different requirements around IN_GUEST_MODE than sending interrupts, e.g. KVM manually processes the IRR on every VM-Enter and so lack of an IPI is a non-issue, whereas missing an IPI for a kick is problematic. In other words, EXITING_GUEST_MODE really needs to mean "existing the run loop".

Do you mean:
“one small part” (as in give this a shot, maybe),
or
“no small part” (as in good-god-don’t-do-this!)

I’m assuming you meant one small part :) sure, how about something like:

To my earlier comment about when to do this within a few instructions, I don’t want to clobber other stuff happening as part of the enter/exit, what if we repurposed/renamed vmx_update_host_rsp and vmx_spec_ctrl_restore_host to make them “do stuff before entry” and “do stuff right after entry returns” functions. That way we wouldn’t have to add another other function calls or change the existing control flow all that much.

In “do before” we could set something like vcpu->non_root_mode = true In “do after” we could set vcpu->non_root_mode = false

Or perhaps something like (respectively)
vcpu->operational_state = NON_ROOT_MODE
vcpu->operational_state = ROOT_MODE

Using the root/non-root moniker would precisely track when the whether the vCPU is in guest, and aligns with the language used to describe such a state from the SDM.

Thoughts?

Quoted 7 lines

E.g. toggle the dedicated flag within a few instructions of VM-Enter and VM-Exit for maximum efficiency for interrupts, and avoid having to make vcpu->mode more complex than it already is.

To add clarity, we could even rename IN_GUEST_MODE and EXITING_GUEST_MODE to something like IN_RUN_LOOP and EXITING_RUN_LOOP.

Yea, this feels like a good idea to reduce confusion. Let’s nail this thread down, then I can propose a wider cleanup?

Quoted 16 lines
Quoted 16 lines

---

arch/x86/kvm/svm/svm.c |  7 +++++++
arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++
arch/x86/kvm/x86.c     |  8 ++++++++
3 files changed, 38 insertions(+)
arch/x86/kvm/svm/svm.c +0 -0
index ce362e88a567..5f0c118a3ffd 100644
@@ -3907,6 +3907,13 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in
 	else
 		__svm_vcpu_run(svm, spec_ctrl_intercepted);
 
Quoted 6 lines

arch/x86/kvm/svm/svm.c static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in · line 3910

Quoted 1 line
arch/x86/kvm/svm/svm.c +1 -0
+	/* Optimize IPI reduction by setting mode immediately after vmexit
	/*
	 * Because KVM isn't the crazy land of net/ block comments should like
	 * this. 
	 */

Ack, will change in v2

Quoted 16 lines
Quoted 16 lines
arch/x86/kvm/svm/svm.c +6 -0
+	 * without a memmory barrier as this as not paired anywhere.
+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
+	 * barrier, after the host is done fully restoring various host states.
+	 */
+	vcpu->mode = EXITING_GUEST_MODE;
+
 	guest_state_exit_irqoff();
 }
 
arch/x86/kvm/vmx/vmx.c +0 -0
index 63247c57c72c..243dcb87c727 100644
@@ -5878,6 +5878,17 @@ static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu)
 
Quoted 9 lines

arch/x86/kvm/vmx/vmx.c static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu) · line 5884

Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +4 -0
 	if (!vmx->req_immediate_exit &&
 	    !unlikely(vmx->loaded_vmcs->hv_timer_soft_disabled)) {
+		/* Reset IN_GUEST_MODE since we're going to reenter
+		 * guest as part of this fast path. This is done as
+		 * an optimization without a memory barrier since
+		 * EXITING_GUEST_MODE is also set without a memory

Heh, justifying the lack of a memory barrier by saying pointing out the other code you added doesn't use a memory barrier is interesting, to put it politely.

Message received! Will mop it up.

Quoted 12 lines

arch/x86/kvm/vmx/vmx.c static fastpath_t handle_fastpath_preemption_timer(struct kvm_vcpu *vcpu) · line 5891

Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +7 -0
+		 * barrier. This also needs to be reset prior to
+		 * calling apic_timer_expired() so that
+		 * kvm_use_posted_timer_interrupt() returns the proper
+		 * value.
+		 */
+		if (vcpu->mode == EXITING_GUEST_MODE)
+			vcpu->mode = IN_GUEST_MODE;

It's far easier, likely more performant, documents why this has a chance of working, and significantly less error prone to do this unconditionally in either assembly or after the EXIT_FASTPATH_REENTER_GUEST check in vcpu_enter_guest().

Ack, will do that there.

#10 lore raw
Quoted 10 lines · Jon Kohler
Quoted 6 lines · Sean Christopherson
Quoted 1 line

Changing vcpu->mode away from IN_GUEST_MODE as early as possible

Except this isn't as early as possible. If we're going to bother doing something like this, my vote is to move it into assembly.

In vmenter.S, tacking on to call vmx_spec_ctrl_restore_host seemed like the most logical place after handling all of the state saves and RSB work. Are you saying put it even closer to the exit, meaning before the FILL_RETURN_BUFFER?

Yes, assuming that's safe, in which case it's truly the "as early as possible" location.

Quoted 25 lines
Quoted 18 lines
Quoted 6 lines

gives IPI senders as much runway as possible to avoid ringing doorbell or sending posted interrupt IPI in AMD and Intel, respectively. Since this is done without an explicit memory barrier, the worst case is that the IPI sender sees IN_GUEST_MODE still and sends a spurious event, which is the behavior prior to this patch.

No, worst case scenario is that kvm_vcpu_exiting_guest_mode() sees EXITING_GUEST_MODE and doesn't kick the vCPU. For "kicks" that set a request, kvm_vcpu_exit_request() will punt the vCPU out of the tight run loop, though there might be ordering issues.

But whether or not there are ordering issues is a moot point since there are uses of kvm_vcpu_kick() that aren't accompanied by a request, e.g. to purge the PML buffers. In other words, kvm_vcpu_kick() absolutely cannot have false negatives. We could modify KVM to require a request when using kvm_vcpu_kick(), but that's a bit of a hack, and all of the possible ordering problems is still a pile of complexity I'd rather avoid.

Quoted 6 lines

No small part of me thinks we'd be better off adding a dedicated flag to very precisely track whether or not the vCPU is truly "in the guest" for the purposes of sending IPIs. Things like kicks have different requirements around IN_GUEST_MODE than sending interrupts, e.g. KVM manually processes the IRR on every VM-Enter and so lack of an IPI is a non-issue, whereas missing an IPI for a kick is problematic. In other words, EXITING_GUEST_MODE really needs to mean "existing the run loop".

Quoted 4 lines

Do you mean:
“one small part” (as in give this a shot, maybe),
or
“no small part” (as in good-god-don’t-do-this!)

Neither. "No small part" as in "Most of my brain", i.e. "I haven't completely thought things through, but I think we'd be better off adding a dedicated flag".

Quoted 7 lines

I’m assuming you meant one small part :) sure, how about something like:

To my earlier comment about when to do this within a few instructions, I don’t want to clobber other stuff happening as part of the enter/exit, what if we repurposed/renamed vmx_update_host_rsp and vmx_spec_ctrl_restore_host to make them “do stuff before entry” and “do stuff right after entry returns” functions. That way we wouldn’t have to add another other function calls or change the existing control flow all that much.

I'd prefer not to wrap vmx_update_host_rsp(), that thing is a very special snowflake.

I don't see why we'd have to add function calls or change the existing control flow anyways. The asm flows for VMX and SVM both take the vCPU in the form of @vmx and @svm, so accessing the proposed excution mode field is just a matter of adding an entry in arch/x86/kvm/kvm-asm-offsets.c.

And now that kvm-asm-offsets.c exists, I think it makes sense to drop the @regs parameter for __vmx_vcpu_run(), e.g. to roughly match __svm_vcpu_run().

With that done as prep, accessing the vCPU immediately before/after VM-Enter and VM-Exit is easy.

As a rough, incomplete sketch for VMX:

arch/x86/kvm/vmx/vmenter.S +13 -4
index 766c6b3ef5ed..f80553e34f26 100644
@@ -102,7 +102,7 @@ SYM_FUNC_START(__vmx_vcpu_run)
         * an LFENCE to stop speculation from skipping the wrmsr.
         */
 
-       /* Load @regs to RAX. */
+       /* Load @vmx to RAX. */
        mov (%_ASM_SP), %_ASM_AX
 
        /* Check if vmlaunch or vmresume is needed */
@@ -125,7 +125,11 @@ SYM_FUNC_START(__vmx_vcpu_run)
        mov VCPU_R14(%_ASM_AX), %r14
        mov VCPU_R15(%_ASM_AX), %r15
 #endif
-       /* Load guest RAX.  This kills the @regs pointer! */
+
+       /* Mark the vCPU as executing in the guest! */
+       movb $IN_GUEST_MODE, VCPU_EXECUTION_MODE(_%ASM_AX)
+
+       /* Load guest RAX.  This kills the @vmx pointer! */
        mov VCPU_RAX(%_ASM_AX), %_ASM_AX
 
        /* Check EFLAGS.ZF from 'testb' above */
@@ -163,9 +167,11 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL)
        /* Temporarily save guest's RAX. */
        push %_ASM_AX
 
-       /* Reload @regs to RAX. */
+       /* Reload @vmx to RAX. */
        mov WORD_SIZE(%_ASM_SP), %_ASM_AX
 
+       movb $IN_HOST_MODE, VCPU_EXECUTION_MODE(_%ASM_AX)
+
        /* Save all guest registers, including RAX from the stack */
        pop           VCPU_RAX(%_ASM_AX)
        mov %_ASM_CX, VCPU_RCX(%_ASM_AX)
@@ -189,9 +195,12 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL)
        xor %ebx, %ebx
 
 .Lclear_regs:
-       /* Discard @regs.  The register is irrelevant, it just can't be RBX. */
+       /* Pop @vmx.  The register can be anything except RBX. */
        pop %_ASM_AX
 
+       /* Set the execution mode (again) in case of VM-Fail. */
+       movb $IN_HOST_MODE, VCPU_EXECUTION_MODE(_%ASM_AX)
+
        /*
         * Clear all general purpose registers except RSP and RBX to prevent
         * speculative use of the guest's values, even those that are reloaded
Quoted 7 lines

In “do before” we could set something like vcpu->non_root_mode = true In “do after” we could set vcpu->non_root_mode = false

Or perhaps something like (respectively)
vcpu->operational_state = NON_ROOT_MODE
vcpu->operational_state = ROOT_MODE

Quoted 5 lines

Using the root/non-root moniker would precisely track when the whether the vCPU is in guest, and aligns with the language used to describe such a state from the SDM.

Thoughts?

No, we should use KVM-defined names, not VMX-defined names, because we'll want the same logic for SVM. If we first rename GUEST_MODE => RUN_LOOP, then we can use IN_GUEST_MODE and IN_HOST_MODE or whatever.

maintainer
#11 lore raw
Quoted 3 lines · Sean Christopherson

And now that kvm-asm-offsets.c exists, I think it makes sense to drop the @regs parameter for __vmx_vcpu_run(), e.g. to roughly match __svm_vcpu_run().

Yeah, that I had already here (https://lore.kernel.org/kvm/[email protected]/T/#me77cdd848fa86cdce03f59447ddb4f2cba45f79c) but then I dropped when it became clear the SVM series was targeting 6.1. I planned to resubmit for 6.3 after the holidays.

Paolo