EFI mixed mode + perf = rampant triple faults
I figured I should send this email before I forget about this issue:
If you run perf record across any EFI mixed mode call or otherwise receive an NMI or MCE, the machine triple-faults. The cause is
straightforward: there is no valid IDT when we have long mode disabled for the duration of the EFI call.
As far as I know, the only way to have continuously functional interrupt handling across a long mode transition is to install an interrupt vector table and hope that CPUs actually do something intelligent when receiving an interrupt with LME=1, LMA=1, and PG=0. Yuck.
Could we get away with issuing 32-bit EFI calls in compat mode, i.e. with a 32-bit CPL0 CS but while still in long mode? I think that delivery of an IST interrupt (which includes both NMI and MCE) will correctly switch to a fully valid 64-bit state and would correctly switch back when we execute IRET at the end. (Am I missing some reason that switching bitness without a privilege level change doesn't work well? I haven't thought of anything, other than the lack of SS controls on intra-ring interrupts, but that shouldn't be an issue here.)
As an added benefit, this would considerably simplify the code.
--Andy
[trying again with .org spelled correctly. also cc: bpetkov]
Quoted 24 lines · Andy Lutomirski
I figured I should send this email before I forget about this issue:
If you run perf record across any EFI mixed mode call or otherwise receive an NMI or MCE, the machine triple-faults. The cause is
straightforward: there is no valid IDT when we have long mode disabled for the duration of the EFI call.As far as I know, the only way to have continuously functional interrupt handling across a long mode transition is to install an interrupt vector table and hope that CPUs actually do something intelligent when receiving an interrupt with LME=1, LMA=1, and PG=0. Yuck.
Could we get away with issuing 32-bit EFI calls in compat mode, i.e. with a 32-bit CPL0 CS but while still in long mode? I think that delivery of an IST interrupt (which includes both NMI and MCE) will correctly switch to a fully valid 64-bit state and would correctly switch back when we execute IRET at the end. (Am I missing some reason that switching bitness without a privilege level change doesn't work well? I haven't thought of anything, other than the lack of SS/SP controls on intra-ring interrupts, but that shouldn't be an issue here.)
As an added benefit, this would considerably simplify the code.
--Andy
Quoted 9 lines · Andy Lutomirski
[trying again with .org spelled correctly. also cc: bpetkov]
Quoted 6 lines · Andy Lutomirski
I figured I should send this email before I forget about this issue:
If you run perf record across any EFI mixed mode call or otherwise receive an NMI or MCE, the machine triple-faults. The cause is
straightforward: there is no valid IDT when we have long mode disabled for the duration of the EFI call.
Right, the lack of IDT is intentional since we disable interrupts while making the EFI call and so far I have side-stepped (ignored) the NMI/MCE issue.
Perf is an interesting use case. I've admittedly never used it with EFI mixed mode, but yes, we should definitely get that working (if NMI/MCE handling wasn't justification enough).
Quoted 14 lines
Quoted 14 lines
As far as I know, the only way to have continuously functional interrupt handling across a long mode transition is to install an interrupt vector table and hope that CPUs actually do something intelligent when receiving an interrupt with LME=1, LMA=1, and PG=0. Yuck.
Could we get away with issuing 32-bit EFI calls in compat mode, i.e. with a 32-bit CPL0 CS but while still in long mode? I think that delivery of an IST interrupt (which includes both NMI and MCE) will correctly switch to a fully valid 64-bit state and would correctly switch back when we execute IRET at the end. (Am I missing some reason that switching bitness without a privilege level change doesn't work well? I haven't thought of anything, other than the lack of SS/SP controls on intra-ring interrupts, but that shouldn't be an issue here.)
Quoted 1 line
Quoted 1 line
As an added benefit, this would considerably simplify the code.
I can't immediately think of a reason that this wouldn't work, but I've Cc'd more x86 folks for additional insight.
I will schedule some time to look into this issue in the new year. Thanks Andy.
Signature · 1 line
Matt Fleming, Intel Open Source Technology Center
Quoted 15 lines · Matt Fleming
Quoted 14 lines · Andy Lutomirski
Quoted 14 lines
As far as I know, the only way to have continuously functional interrupt handling across a long mode transition is to install an interrupt vector table and hope that CPUs actually do something intelligent when receiving an interrupt with LME=1, LMA=1, and PG=0. Yuck.
Could we get away with issuing 32-bit EFI calls in compat mode, i.e. with a 32-bit CPL0 CS but while still in long mode? I think that delivery of an IST interrupt (which includes both NMI and MCE) will correctly switch to a fully valid 64-bit state and would correctly switch back when we execute IRET at the end. (Am I missing some reason that switching bitness without a privilege level change doesn't work well? I haven't thought of anything, other than the lack of SS/SP controls on intra-ring interrupts, but that shouldn't be an issue here.)
Quoted 7 lines
Quoted 1 line
Quoted 1 line
As an added benefit, this would considerably simplify the code.
I can't immediately think of a reason that this wouldn't work, but I've Cc'd more x86 folks for additional insight.
I will schedule some time to look into this issue in the new year. Thanks Andy.
I finally got some time to look into this, and running with __KERNEL32_CS seems to work fine at runtime both with Qemu + 32-bit OVMF and on my ASUS T100. Manually triggering an MCE exception immediately before invoking the firmware service recovers gracefully.
Where this won't work so well is at boot time before we jump to the kernel proper. There, we still need to restore the firmware's GDT so that interrupts are serviced correctly before ExitBootServices() (in particular, ia32 Tianocore assumes __KERNEL_CS is a 32-bit CS).
Which means the code to handle mixed mode calls at boot time and runtime has now diverged. Fixing that is probably just a SMOP to maximise code reuse though.
I'll post a patch after some more testing.
Signature · 1 line
Matt Fleming, Intel Open Source Technology Center
Quoted 21 lines · Matt Fleming
Quoted 20 lines · Matt Fleming
Quoted 14 lines · Andy Lutomirski
Quoted 14 lines
As far as I know, the only way to have continuously functional interrupt handling across a long mode transition is to install an interrupt vector table and hope that CPUs actually do something intelligent when receiving an interrupt with LME=1, LMA=1, and PG=0. Yuck.
Could we get away with issuing 32-bit EFI calls in compat mode, i.e. with a 32-bit CPL0 CS but while still in long mode? I think that delivery of an IST interrupt (which includes both NMI and MCE) will correctly switch to a fully valid 64-bit state and would correctly switch back when we execute IRET at the end. (Am I missing some reason that switching bitness without a privilege level change doesn't work well? I haven't thought of anything, other than the lack of SS/SP controls on intra-ring interrupts, but that shouldn't be an issue here.)
Quoted 1 line
Quoted 1 line
As an added benefit, this would considerably simplify the code.
I can't immediately think of a reason that this wouldn't work, but I've Cc'd more x86 folks for additional insight.
Quoted 7 lines
Quoted 2 lines
I will schedule some time to look into this issue in the new year. Thanks Andy.
I finally got some time to look into this, and running with
__KERNEL32_CSseems to work fine at runtime both with Qemu + 32-bit OVMF and on my ASUS T100. Manually triggering an MCE exception immediately before invoking the firmware service recovers gracefully.
How are you manually triggering an MCE? I've been playing with some MCE stuff recently, but the only reasonably reliable way I know of to trigger an MCE is using WHEA, and I don't have a box with WHEA, and I assume your ASUS T100 doesn't either.
Quoted 5 lines
Where this won't work so well is at boot time before we jump to the kernel proper. There, we still need to restore the firmware's GDT so that interrupts are serviced correctly before
ExitBootServices()(in particular, ia32 Tianocore assumes__KERNEL_CSis a 32-bit CS).
Tianocore makes assumptions about the kernel's GDT layout? Yuck.
--Andy
Quoted 4 lines · Andy Lutomirski
How are you manually triggering an MCE? I've been playing with some MCE stuff recently, but the only reasonably reliable way I know of to trigger an MCE is using WHEA, and I don't have a box with WHEA, and I assume your ASUS T100 doesn't either.
asm volatile("int $18");
Quoted 1 line
Tianocore makes assumptions about the kernel's GDT layout? Yuck.
Yuck indeed.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Andy Lutomirski
How are you manually triggering an MCE? I've been playing with some MCE stuff recently, but the only reasonably reliable way I know of to trigger an MCE is using WHEA, and I don't have a box with WHEA, and I assume your ASUS T100 doesn't either.
asm
volatile("int $18");
That's not a real MCE, though -- it happens synchronously instead of at MCE priority with all the associated messiness. Or is the idea to just stick that in after switching to the 32-bit mode being tested?
--Andy
Quoted 11 lines
Quoted 2 lines
Tianocore makes assumptions about the kernel's GDT layout? Yuck.
Yuck indeed.
--
Regards/Gruss,
Boris.Sent from a fat crate under my desk. Formatting is fine. --
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC
Quoted 1 line · Andy Lutomirski
That's not a real MCE, though -- it happens synchronously instead of
MCE can be synchronous in a sense too, as a result of executing an insn, for example, i.e., EIPV bit set.
Quoted 2 lines
at MCE priority with all the associated messiness. Or is the idea to just stick that in after switching to the 32-bit mode being tested?
Yes, the idea was to simply trigger a real exception after switching to 32-bit while still with 64-bit IDT handlers.
Signature · 5 lines
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--Quoted 12 lines · Borislav Petkov
Quoted 1 line · Andy Lutomirski
That's not a real MCE, though -- it happens synchronously instead of
MCE can be synchronous in a sense too, as a result of executing an insn, for example, i.e., EIPV bit set.
Quoted 2 lines
at MCE priority with all the associated messiness. Or is the idea to just stick that in after switching to the 32-bit mode being tested?
Yes, the idea was to simply trigger a real exception after switching to 32-bit while still with 64-bit IDT handlers.
Seems like a reasonable test to me.
--Andy
Quoted 5 lines · Andy Lutomirski
How are you manually triggering an MCE? I've been playing with some MCE stuff recently, but the only reasonably reliable way I know of to trigger an MCE is using WHEA, and I don't have a box with WHEA, and I assume your ASUS T100 doesn't either.
As Borislav mentions, I used 'int $18', solely to trigger the 64-bit exception handler code paths in the middle of the EFI mixed mode code.
Quoted 6 lines
Quoted 4 lines
Where this won't work so well is at boot time before we jump to the kernel proper. There, we still need to restore the firmware's GDT so that interrupts are serviced correctly before
ExitBootServices()(in particular, ia32 Tianocore assumes__KERNEL_CSis a 32-bit CS).Tianocore makes assumptions about the kernel's GDT layout? Yuck.
No, but 32-bit Tianocore does rely on the second GDT entry being a 32-bit CS.
It has no knowledge of Linux's GDT layout.
Signature · 1 line
Matt Fleming, Intel Open Source Technology Center
Quoted 8 lines · Matt Fleming
Quoted 2 lines
Tianocore makes assumptions about the kernel's GDT layout? Yuck.
No, but 32-bit Tianocore does rely on the second GDT entry being a 32-bit CS.
It has no knowledge of Linux's GDT layout.
If it assumes that descriptor 16 is a 32-bit CS (and what about data? 24?) that *is* making assumptions on the kernel.
-hpa
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Matt Fleming
Quoted 2 lines
Tianocore makes assumptions about the kernel's GDT layout? Yuck.
Quoted 8 lines
Quoted 5 lines
No, but 32-bit Tianocore does rely on the second GDT entry being a 32-bit CS.
It has no knowledge of Linux's GDT layout.
If it assumes that descriptor 16 is a 32-bit CS (and what about data? 24?) that *is* making assumptions on the kernel.
Bear in mind that this is before ExitBootServices() is invoked, so where the firmware still thinks it (not the OS) "owns" the platform.
None of this comes into play at Runtime.
Signature · 1 line
Matt Fleming, Intel Open Source Technology Center