VMX: Host NMI triggering on NMI vmexit

DISCUSSION Jan Kiszka 4 participants 9 messages Sep 16, 2008 – Sep 23, 2008

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

Jan Kiszka

author

2008-09-16 11:26 UTC #0 lore raw

Sheng,

out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if the VM exited due to an external NMI event. According to Intel specs I have, software-triggered NMIs do not block hardware NMIs. So are we facing the risk to receive another NMI while running the first handler? Or will the VM be left with the hardware blocking logic armed? Or does Linux not care about NMI handler re-entrance?

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Avi Kivity

2008-09-19 22:37 UTC #1 lore raw

Quoted 9 lines · Jan Kiszka

Sheng,
out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if the VM exited due to an external NMI event. According to Intel specs I have, software-triggered NMIs do not block hardware NMIs. So are we facing the risk to receive another NMI while running the first handler? Or will the VM be left with the hardware blocking logic armed? Or does Linux not care about NMI handler re-entrance?

All good questions. Usually this doesn't happen since NMI sources are far apart (oprofile, watchdog).

Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.

Signature · 2 lines

I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Jan Kiszka

2008-09-20 06:56 UTC #2 lore raw

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Jan Kiszka
Sheng,
out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if the VM exited due to an external NMI event. According to Intel specs I have, software-triggered NMIs do not block hardware NMIs. So are we facing the risk to receive another NMI while running the first handler? Or will the VM be left with the hardware blocking logic armed? Or does Linux not care about NMI handler re-entrance?

Quoted 2 lines

All good questions. Usually this doesn't happen since NMI sources are far apart (oprofile, watchdog).

Only true until you have multiple unsynchronized NMI sources, e.g. inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs in kvm's and my own NMI code that were triggered by such a scenario (sigh...).

Quoted 4 lines

Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.

That sounds like a good idea, will look into this right after fixing the other NMI issues.

Jan

Jan Kiszka

author

2008-09-22 10:54 UTC #3 lore raw

In reply to Avi Kivity

Quoted 11 lines · Avi Kivity

Quoted 9 lines · Jan Kiszka
Sheng,
out of curiosity: vmx_vcpu_run invokes 'int $2' to trigger a host NMI if the VM exited due to an external NMI event. According to Intel specs I have, software-triggered NMIs do not block hardware NMIs. So are we facing the risk to receive another NMI while running the first handler? Or will the VM be left with the hardware blocking logic armed? Or does Linux not care about NMI handler re-entrance?

Quoted 5 lines

All good questions. Usually this doesn't happen since NMI sources are far apart (oprofile, watchdog).
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.

Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service? And is it safe to assume VMX == LAPIC available and usable?

However, this is how it would look like. Yet untested, /me has to replace his host kernel first...

Jan

---
 arch/x86/kernel/genapic_64.c |    1 +
 arch/x86/kernel/ipi.c        |    1 +
 arch/x86/kvm/vmx.c           |    6 +++---
 3 files changed, 5 insertions(+), 3 deletions(-)

arch/x86/kernel/genapic_64.c +1 -0

Index: b/arch/x86/kernel/genapic_64.c
===================================================================
@@ -65,6 +65,7 @@ void send_IPI_self(int vector)
 {
 	__send_IPI_shortcut(APIC_DEST_SELF, vector, APIC_DEST_PHYSICAL);
 }
+EXPORT_SYMBOL_GPL(send_IPI_self);
 
 int __init acpi_madt_oem_check(char *oem_id, char *oem_table_id)
 {

arch/x86/kernel/ipi.c +1 -0

Index: b/arch/x86/kernel/ipi.c
===================================================================
@@ -77,6 +77,7 @@ void send_IPI_self(int vector)
 {
 	__send_IPI_shortcut(APIC_DEST_SELF, vector);
 }
+EXPORT_SYMBOL_GPL(send_IPI_self);
 
 /*
  * This is used to send an IPI with no shorthand notation (the destination is

arch/x86/kvm/vmx.c +3 -3

Index: b/arch/x86/kvm/vmx.c
===================================================================
@@ -2488,7 +2488,7 @@ static int handle_exception(struct kvm_v
 		set_bit(irq / BITS_PER_LONG, &vcpu->arch.irq_summary);
 	}
 
-	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == 0x200) /* nmi */
+	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR)
 		return 1;  /* already handled by vmx_vcpu_run() */
 
 	if (is_no_device(intr_info)) {

@@ -3321,10 +3321,10 @@ static void vmx_vcpu_run(struct kvm_vcpu
 	intr_info = vmcs_read32(VM_EXIT_INTR_INFO);
 
 	/* We need to handle NMIs before interrupts are enabled */
-	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == 0x200 &&
+	if ((intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
 	    (intr_info & INTR_INFO_VALID_MASK)) {
 		KVMTRACE_0D(NMI, vcpu, handler);
-		asm("int $2");
+		send_IPI_self(APIC_DM_NMI);
 	}
 
 	vmx_complete_interrupts(vmx);

Avi Kivity

2008-09-22 11:00 UTC #4 lore raw

Quoted 6 lines · Jan Kiszka

Quoted 3 lines
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.
Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service?

If we can make a case for it (I think we can), then I don't see why not.

Sheng, can you confirm that 'int 2' is problematic, and that nmi-via-lapic is the best workaround?

Quoted 4 lines

And is it safe to
assume VMX == LAPIC available and usable?

Yes.

Quoted 1 line

However, this is how it would look like.

I'd define a send_nmi_self() instead, to allow the implementation to change (x2apic/etc).

Quoted 3 lines

Yet untested, /me has to
replace his host kernel first...

You could test it in a VM, if someone implements nested vmx :)

btw, looks like svm is not affected by this.

Signature · 1 line

error compiling committee.c: too many arguments to function

Yang, Sheng

2008-09-23 05:33 UTC #5 lore raw

Quoted 11 lines · On Monday 22 September 2008 19:00:38 Avi Kivity wrote:

Quoted 5 lines · Jan Kiszka
Quoted 2 lines
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.
Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service?
If we can make a case for it (I think we can), then I don't see why not.
Sheng, can you confirm that 'int 2' is problematic, and that nmi-via-lapic is the best workaround?

Just back from vacation... :)

Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's too obviously as a hardware issue when using with NMI exiting=1 in vmx nonroot mode, so I have checked it with my colleague, finally found these in SDM 3B 23-2:

The following bullets detail when architectural state is and is not updated in response to VM exits:
• If an event causes a VM exit *directly*, it does not update architectural
state as it would have if it had it not caused the VM exit:
[...]
— *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit
completes.

So we needn't worry about that, and this shouldn't cause any trouble AFAIK...

Jan, seems we need to do more investigating on the issues you met...

Signature · 2 lines

regards
Yang, Sheng

Quoted 19 lines

Quoted 2 lines
And is it safe to
assume VMX == LAPIC available and usable?
Yes.
Quoted 1 line
However, this is how it would look like.
I'd define a send_nmi_self() instead, to allow the implementation to change (x2apic/etc).
Quoted 2 lines
Yet untested, /me has to
replace his host kernel first...
You could test it in a VM, if someone implements nested vmx :)
btw, looks like svm is not affected by this.
--
error compiling committee.c: too many arguments to function

Jan Kiszka

author

2008-09-23 08:48 UTC #6 lore raw

Quoted 26 lines · Sheng

Quoted 9 lines · On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
Quoted 4 lines · Jan Kiszka
Quoted 2 lines
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.
Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service?
If we can make a case for it (I think we can), then I don't see why not.
Sheng, can you confirm that 'int 2' is problematic, and that nmi-via-lapic is the best workaround?
Just back from vacation... :)
Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's too obviously as a hardware issue when using with NMI exiting=1 in vmx nonroot mode, so I have checked it with my colleague, finally found these in SDM 3B 23-2:
The following bullets detail when architectural state is and is not updated in response to VM exits:
• If an event causes a VM exit *directly*, it does not update architectural
state as it would have if it had it not caused the VM exit:
[...]
— *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit
completes.

Quoted 1 line

So we needn't worry about that, and this shouldn't cause any trouble AFAIK...

Fine, problems--. :)

Quoted 2 lines

Jan, seems we need to do more investigating on the issues you met...

Sorry, which one do you mean now?

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Yang, Sheng

2008-09-23 08:56 UTC #7 lore raw

Quoted 29 lines · On Tuesday 23 September 2008 16:47:54 Jan Kiszka wrote:

Quoted 28 lines · Sheng
Quoted 11 lines · On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
Quoted 5 lines · Jan Kiszka
Quoted 2 lines
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.
Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service?
If we can make a case for it (I think we can), then I don't see why not.
Sheng, can you confirm that 'int 2' is problematic, and that nmi-via-lapic is the best workaround?
Just back from vacation... :)
Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's too obviously as a hardware issue when using with NMI exiting=1 in vmx nonroot mode, so I have checked it with my colleague, finally found these in SDM 3B 23-2:
The following bullets detail when architectural state is and is not updated in response to VM exits:
• If an event causes a VM exit *directly*, it does not update
architectural state as it would have if it had it not caused the VM exit:
[...]
— *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit completes.

Quoted 8 lines

Quoted 2 lines
So we needn't worry about that, and this shouldn't cause any trouble AFAIK...
Fine, problems--. :)
Quoted 1 line
Jan, seems we need to do more investigating on the issues you met...
Sorry, which one do you mean now?

You said
"Only true until you have multiple unsynchronized NMI sources, e.g. inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs in kvm's and my own NMI code that were triggered by such a scenario (sigh...)." ?

If it's not related to this one. That's fine. :)

Signature · 2 lines

regards
Yang, Sheng

Quoted 6 lines

Jan
--
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Jan Kiszka

author

2008-09-23 09:01 UTC #8 lore raw

Quoted 34 lines · Sheng

Quoted 26 lines · On Tuesday 23 September 2008 16:47:54 Jan Kiszka wrote:
Quoted 25 lines · Sheng
Quoted 9 lines · On Monday 22 September 2008 19:00:38 Avi Kivity wrote:
Quoted 4 lines · Jan Kiszka
Quoted 2 lines
Maybe the answer is to generate the local nmi via an IPI-to-self command to the local apic.
Going this way leaves me with a few questions: Will it be OK for the related mainainers to export the required service?
If we can make a case for it (I think we can), then I don't see why not.
Sheng, can you confirm that 'int 2' is problematic, and that nmi-via-lapic is the best workaround?
Just back from vacation... :)
Jan said is true, "int 2" itself won't block subsequent NMIs. But I think it's too obviously as a hardware issue when using with NMI exiting=1 in vmx nonroot mode, so I have checked it with my colleague, finally found these in SDM 3B 23-2:
The following bullets detail when architectural state is and is not updated in response to VM exits:
• If an event causes a VM exit *directly*, it does not update
architectural state as it would have if it had it not caused the VM exit:
[...]
— *An NMI causes subsequent NMIs to be blocked*, but only after the VM exit completes.
Quoted 6 lines
Quoted 2 lines
So we needn't worry about that, and this shouldn't cause any trouble AFAIK...
Fine, problems--. :)
Quoted 1 line
Jan, seems we need to do more investigating on the issues you met...
Sorry, which one do you mean now?

Quoted 7 lines

You said
"Only true until you have multiple unsynchronized NMI sources, e.g. inter-CPU NMIs of kgdb + a watchdog. I just stumbled over several bugs in kvm's and my own NMI code that were triggered by such a scenario (sigh...)." ?
If it's not related to this one. That's fine. :)

Nope, the actual issues were related to guests facing "NMI storms" (and should be fixed by my series). That made me wonder what would happen on the host side if int 2 wasn't safe. But it is, as we know now.

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux