VMX: NMI injection without virtual NMI support

Hi Sheng,

we had parts of this discussion privately a while back, but now I need to dig deeper. I finally have to complete and push out user space NMI injection that bitrots over and over again in my queue.

Intel CPUs with virtual NMI support can easily be programmed to inject NMIs only when the guest CPU is ready or tell the guest to exit as soon as it changes its NMI readiness. But we are also facing a lot of CPUs that were produced without this feature, and we need to find some way to achieve virtual NMI support for them, even if the interruptibility of the guest is not as good as with true virtual NMIs.

Now my questions to you or anyone else with VMX expert knowledge:

The System Programming Guide, Volume 3B, 27.2 says that "Without NMI-window exiting support, the VMM will need to poll and check the interruptibility state of the guest to deliver virtual NMIs." Interruptibility involves, besides "blocked by MOV SS" and "blocked by STI", the NMI nesting. So, how can the VMM track if the guest is still inside its NMI handler after event injection?

According to 22.6.1, NMI blocking is active as long as the guest runs if the "NMI-exiting" control bit is 1 (and setting it appears to be required to avoid that the guest can block NMIs for the host, right?).

But what does this mean for bit 3 of the interruptibility state? Can I still use it after some guest exit for whatever reason (like a hard IRQ) to poll if the guest can now accept pending virtual NMIs?

In that case, virtual NMI injections would only be delayed a bit on older CPUs, but could still work reliably, right? Or is there some other way to track the NMI interruptibility on such CPUs?

TiA,
Jan

Quoted 13 lines · On Thursday 11 September 2008 23:11:14 Jan Kiszka wrote:

Hi Sheng,

we had parts of this discussion privately a while back, but now I need to dig deeper. I finally have to complete and push out user space NMI injection that bitrots over and over again in my queue.

Intel CPUs with virtual NMI support can easily be programmed to inject NMIs only when the guest CPU is ready or tell the guest to exit as soon as it changes its NMI readiness. But we are also facing a lot of CPUs that were produced without this feature, and we need to find some way to achieve virtual NMI support for them, even if the interruptibility of the guest is not as good as with true virtual NMIs.

Quoted 8 lines

Now my questions to you or anyone else with VMX expert knowledge:

The System Programming Guide, Volume 3B, 27.2 says that "Without NMI-window exiting support, the VMM will need to poll and check the interruptibility state of the guest to deliver virtual NMIs." Interruptibility involves, besides "blocked by MOV SS" and "blocked by STI", the NMI nesting. So, how can the VMM track if the guest is still inside its NMI handler after event injection?

I think, we have no direct idea about that...

Quoted 3 lines

According to 22.6.1, NMI blocking is active as long as the guest runs if the "NMI-exiting" control bit is 1 (and setting it appears to be required to avoid that the guest can block NMIs for the host, right?).

Yes.

Quoted 3 lines

But what does this mean for bit 3 of the interruptibility state? Can I still use it after some guest exit for whatever reason (like a hard IRQ) to poll if the guest can now accept pending virtual NMIs?

No. Though the public spec is a little ambiguous, I just checked some document and found that, if "virtual NMI" is not enabled, "Block by NMI" just indicated block *host physical* NMI rather than virtual NMI.

Quoted 4 lines

In that case, virtual NMI injections would only be delayed a bit on older CPUs, but could still work reliably, right? Or is there some other way to track the NMI interruptibility on such CPUs?

I hope so, but sadly seems we can't know if we can inject NMI to guest without enable "virtual NMI".

After some brainstorming with my colleague Haitao, we found one way *may* can work on non-virtual nmi supported hardware, but it's very tricky and untested(just a theoretical idea), also sacrifice host physical NMI, and depends on guest NMI implement.

The basic idea is:
1. Disable "NMI Exiting" feature, so that guest would handle any host physical NMI.
2. Set "Blocking by NMI" to true. So if guest execute "iret", this bit should be cleaned. This also block physical NMIs.
3. Enable "IRQ Window" and ensure IDT vector 2 is a interrupt gate(so that IF bit is cleared when executing the handler). So we use IRQ window to replace NMI window here, otherwise for the period between IRET and VMEXIT, host physical NMIs would be handled by guest handler.
4. Check "IRQ Window" exit, if "Blocking by NMI" is cleared(also STI and SS), we can inject NMI.

It's just untested idea, and I think it's quite tricky. If you still have interest, you may try it, but I can't guarantee the result...

Well, I thought in this direction already as well. But I wasn't sure if, while the guest is in NMI context, hard IRQs will also be blocked and won't cause guest exists anymore. Can you comment on this?

However, even if that is no issue, I do not really like this workaround. Specifically the need to fiddle with the guest's IDT and, of course, that we may delays host NMIs.

I'm now playing with this idea, basically a "light" version of yours:
After we injected an NMI, consider the guest being in NMI context until the next IRQ window opens. That may cause lost NMIs if the guest blocks IRQ delivery infinitely, but I would say this is rather untypical and still much better than the current situation (no NMIs at all!). And it is easier to implement. Comments?

Quoted 8 lines · Jan Kiszka

Well, I thought in this direction already as well. But I wasn't sure if, while the guest is in NMI context, hard IRQs will also be blocked and won't cause guest exists anymore. Can you comment on this?

However, even if that is no issue, I do not really like this workaround. Specifically the need to fiddle with the guest's IDT and, of course, that we may delays host NMIs.

Quoted 9 lines

I'm now playing with this idea, basically a "light" version of yours:
After we injected an NMI, consider the guest being in NMI context until the next IRQ window opens. That may cause lost NMIs if the guest blocks IRQ delivery infinitely, but I would say this is rather untypical and still much better than the current situation (no NMIs at all!). And it is easier to implement. Comments?

In some cases misbehaving NMIs are worse than no NMIs. For example, a software watchdog may use NMIs to monitor a system. But if the guest spins with interrupts disabled, the irq window will never open, and NMIs will never be delivered, so the watchdog will deliver a false negative.

Quoted 6 lines

In some cases misbehaving NMIs are worse than no NMIs. For example, a software watchdog may use NMIs to monitor a system. But if the guest spins with interrupts disabled, the irq window will never open, and NMIs will never be delivered, so the watchdog will deliver a false negative.

I fail to see the regression. Currently that watchdog would _always_ deliver false positives and pull the trigger immediately (in fact, this is precisely the situation we face @work with some special board emulation where we have to provide an NMI-based watchdog).

Moreover, only the second and succeeding NMIs under the same interrupts-disabled window need to get lost: Along with injecting the first NMI we could request the IRQ window unconditionally, using it to reset the virtual NMI-blocked state.