x86_64: Make int3 non-magical

This will allow IRQ stacks to nest inside NMIs or similar entries that can happen during IRQ stack setup or teardown.

The Xen code here has a confusing comment.

The new macros won't work correctly if they're invoked with IRQs on. Add a check under CONFIG_DEBUG_ENTRY to detect that.

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/Kconfig.debug       |  2 -
 arch/x86/entry/entry_64.S    | 89 +++++++++++++++++++++++++++++---------------
 arch/x86/kernel/process_64.c |  4 ++
 3 files changed, 63 insertions(+), 32 deletions(-)

Quoted 8 lines · Andy Lutomirski

This will allow IRQ stacks to nest inside NMIs or similar entries that can happen during IRQ stack setup or teardown.

The Xen code here has a confusing comment.

The new macros won't work correctly if they're invoked with IRQs on. Add a check under CONFIG_DEBUG_ENTRY to detect that.

Quoted 6 lines

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/Kconfig.debug       |  2 -
 arch/x86/entry/entry_64.S    | 89 +++++++++++++++++++++++++++++---------------
 arch/x86/kernel/process_64.c |  4 ++
 3 files changed, 63 insertions(+), 32 deletions(-)

What does that apply against?

I get conflicts against current Linus master:

This will allow IRQ stacks to nest inside NMIs or similar entries that can happen during IRQ stack setup or teardown.

The Xen code here has a confusing comment.

The new macros won't work correctly if they're invoked with IRQs on. Add a check under CONFIG_DEBUG_ENTRY to detect that.

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/Kconfig.debug       |  2 -
 arch/x86/entry/entry_64.S    | 89 +++++++++++++++++++++++++++++---------------
 arch/x86/kernel/process_64.c |  4 ++
 3 files changed, 63 insertions(+), 32 deletions(-)

This will allow IRQ stacks to nest inside NMIs or similar entries that can happen during IRQ stack setup or teardown.

The Xen code here has a confusing comment.

The new macros won't work correctly if they're invoked with IRQs on. Add a check under CONFIG_DEBUG_ENTRY to detect that.

+/*
+ * Enters the IRQ stack if we're not already using it. NMI-safe. Clobbers
+ * flags and puts old RSP into old_rsp, and leaves all other GPRs alone.
+ * Requires kernel GSBASE.
+ *
+ * The invariant is that, if irq_count != 1, then the IRQ stack is in use.
+ */

Quoted 9 lines · Andy Lutomirski

This will allow IRQ stacks to nest inside NMIs or similar entries that can happen during IRQ stack setup or teardown.

The Xen code here has a confusing comment.

The new macros won't work correctly if they're invoked with IRQs on. Add a check under CONFIG_DEBUG_ENTRY to detect that.

Quoted 7 lines

+/*
+ * Enters the IRQ stack if we're not already using it. NMI-safe. Clobbers
+ * flags and puts old RSP into old_rsp, and leaves all other GPRs alone.
+ * Requires kernel GSBASE.
+ *
+ * The invariant is that, if irq_count != 1, then the IRQ stack is in use.
+ */

I might be wrong, but shouldn't be this read as 'if irq_count != -1' ?

We don't specifically need IST for things like kprobes, but we do want to avoid rare, surprising extra stack usage if a kprobe hits with a deep stack.

Teach idtentry to use the IRQ stack for selected entries.

This implementation uses the IRQ stack even if we entered from user mode. This disallows tricks like ist_begin_non_atomic. If we ever need such a trick in one of these entries, we can rework this. For now, let's keep it simple.

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/entry/entry_64.S | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

There's nothing IST-worthy about #BP/int3. We don't allow kprobes in the small handful of places in the kernel that run at CPL0 with an invalid stack, and 32-bit kernels have used normal interrupt gates for #BP forever.

Furthermore, we don't allow kprobes in places that have usergs while in kernel mode, so "paranoid" is also unnecessary.

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/entry/entry_64.S |  2 +-
 arch/x86/kernel/traps.c   | 26 +++++++++++++-------------
 2 files changed, 14 insertions(+), 14 deletions(-)

Quoted 13 lines · Andy Lutomirski

There's nothing IST-worthy about #BP/int3. We don't allow kprobes in the small handful of places in the kernel that run at CPL0 with an invalid stack, and 32-bit kernels have used normal interrupt gates for #BP forever.

Furthermore, we don't allow kprobes in places that have usergs while in kernel mode, so "paranoid" is also unnecessary.

Signed-off-by Andy Lutomirski <[email protected]>

---
 arch/x86/entry/entry_64.S |  2 +-
 arch/x86/kernel/traps.c   | 26 +++++++++++++-------------
 2 files changed, 14 insertions(+), 14 deletions(-)

...

Quoted 4 lines

arch/x86/kernel/traps.c +0 -0

@@ -494,7 +494,15 @@ dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code)
      if (poke_int3_handler(regs))
              return;
 

arch/x86/kernel/traps.c dotraplinkage void notrace do_int3(struct pt_regs *regs, long error_code) · line 504

Quoted 8 lines

arch/x86/kernel/traps.c +7 -0

+     /*
+      * Use ist_enter despite the fact that we don't use an IST stack.
+      * We can be called from a kprobe in non-CONTEXT_KERNEL kernel
+      * mode or even during context tracking state changes.
+      *
+      * This means that we can't schedule.  That's okay.
+      */
      ist_enter(regs);

Let's rename that thing. Call it atomic_ctxt_enter or whatever...

OK if I do that as a follow-up? It would probably want to be a separate patch anyway.

Hmm, I'm starting to like this new regime in which we never ever switch to user mode from anywhere other than the standard kernel stack. It looks like even Xen may play along and do it cleanly soon :) Maybe I'll even add an assertion somewhere to make sure we don't break it. (I think this also means that the bad iret fixup can be simplified.)

Also, with all this stuff applied (and the modify_ldt thing, once the Xen folks figure out what's wrong), I think we can reinstate the old LARL check for 16-bit segments and thus prevent naughty users from banging on espfix using only sigreturn.