x86/entry: disallow #DB more
Subsystems (5)
Peter Zijlstra
maintainercover x86/entry: disallow #DB more
These patches disallow #DB during NMI/#MC and allow removing a lot of fugly code.
Patch #6 should probably wait until we've got the KGDB situation sorted because applying that makes a bad situation worse.
Peter Zijlstra
maintainerPATCH 1/6 x86/entry: Introduce local_db_{save,restore}()
In order to allow other exceptions than #DB to disable breakpoints, provide a common helper.
--- arch/x86/include/asm/debugreg.h | 25 +++++++++++++++++++++++++ arch/x86/kernel/traps.c | 18 ++---------------- 2 files changed, 27 insertions(+), 16 deletions(-)
arch/x86/include/asm/debugreg.h +25 -0
@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 ); +/* +* Ensure the compiler doesn't lower the above statements into +* the critical section; disabling breakpoints late would not +* be good. +*/ +barrier (); +} + +static __always_inline void local_db_restore(unsigned long dr7) +{ +/* +* Ensure the compiler doesn't raise this statement into +* the critical section; enabling breakpoints early would +* not be good. +*/ +barrier (); +if (dr7) +set_debugreg (dr7,7 ); +} +#ifdef CONFIG_CPU_SUP_AMD extern void set_dr_addr_mask (unsigned long mask,int dr);#else
arch/x86/kernel/traps.c +2 -16
@@ -727,15 +727,7 @@ static __always_inline void debug_enter(* Entry text is excluded for HW_BP_X and cpu_entry_area, which * includes the entry stack is excluded for everything. */ -get_debugreg (*dr7,7 ); -set_debugreg (0 ,7 ); - -/* -* Ensure the compiler doesn't lower the above statements into -* the critical section; disabling breakpoints late would not -* be good. -*/ -barrier (); +local_db_save (dr7);/* * The Intel SDM says:
@@ -756,13 +748,7 @@ static __always_inline void debug_enter( static __always_inline void debug_exit(unsigned long dr7) { -/* -* Ensure the compiler doesn't raise this statement into -* the critical section; enabling breakpoints early would -* not be good. -*/ -barrier (); -set_debugreg (dr7,7 ); +local_db_restore (dr7); }/*
Re: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 120
Quoted 11 lines · Peter Zijlstra
arch/x86/include/asm/debugreg.h +5
-0@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 );
%dr7 has an architecturally stuck bit in it.
You want *dr7 != 0x400 to avoid writing 0 unconditionally.
Also, API wise, wouldn't it be nicer to write "dr7 = local_db_save()" rather than having a void function returning a single long via pointer?
~Andrew
Peter Zijlstra
maintainerRe: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
Quoted 15 lines · Andrew Cooper
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 120
Quoted 11 lines · Peter Zijlstra
arch/x86/include/asm/debugreg.h +5
-0@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 );%dr7 has an architecturally stuck bit in it.
Quoted 1 line
You want *dr7 !=
0x400to avoid writing 0 unconditionally.
Do we have to have that bit set when writing it? Otherwise I might actually prefer masking it out.
Quoted 2 lines
Also, API wise, wouldn't it be nicer to write "dr7 =
local_db_save()" rather than having a void function returning a single long via pointer?
Probably.. I started with local_irq_save() and .. well, n/m. I'll change it ;-)
Peter Zijlstra
maintainerRe: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Andrew Cooper
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 120
Quoted 11 lines · Peter Zijlstra
arch/x86/include/asm/debugreg.h +5
-0@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 );%dr7 has an architecturally stuck bit in it.
Quoted 4 lines
Quoted 1 line
You want *dr7 !=
0x400to avoid writing 0 unconditionally.Do we have to have that bit set when writing it? Otherwise I might actually prefer masking it out.
I'm an idiot, we write a plain 9..
Quoted 5 lines
Quoted 2 lines
Also, API wise, wouldn't it be nicer to write "dr7 =
local_db_save()" rather than having a void function returning a single long via pointer?Probably.. I started with
local_irq_save()and .. well, n/m. I'll change it ;-)
How's this?
---
arch/x86/include/asm/debugreg.h +30 -0
@@ -113,6 +113,36 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline unsigned long local_db_save (void ) +{ +unsigned long dr7; + +get_debugreg (&dr7,7 ); + dr7 ^=0x400 ; +if (dr7) +set_debugreg (0 ,7 ); +/* +* Ensure the compiler doesn't lower the above statements into +* the critical section; disabling breakpoints late would not +* be good. +*/ +barrier (); + +return dr7; +} + +static __always_inline void local_db_restore(unsigned long dr7) +{ +/* +* Ensure the compiler doesn't raise this statement into +* the critical section; enabling breakpoints early would +* not be good. +*/ +barrier (); +if (dr7) +set_debugreg (dr7,7 ); +} +#ifdef CONFIG_CPU_SUP_AMD extern void set_dr_addr_mask (unsigned long mask,int dr);#else
arch/x86/kernel/traps.c +2 -16
@@ -727,15 +727,7 @@ static __always_inline void debug_enter(* Entry text is excluded for HW_BP_X and cpu_entry_area, which * includes the entry stack is excluded for everything. */ -get_debugreg (*dr7,7 ); -set_debugreg (0 ,7 ); - -/* -* Ensure the compiler doesn't lower the above statements into -* the critical section; disabling breakpoints late would not -* be good. -*/ -barrier (); + *dr7 =local_db_save ();/* * The Intel SDM says:
@@ -756,13 +748,7 @@ static __always_inline void debug_enter( static __always_inline void debug_exit(unsigned long dr7) { -/* -* Ensure the compiler doesn't raise this statement into -* the critical section; enabling breakpoints early would -* not be good. -*/ -barrier (); -set_debugreg (dr7,7 ); +local_db_restore (dr7); }/*
Re: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
Quoted 40 lines · Peter Zijlstra
Quoted 16 lines · Peter Zijlstra
Quoted 16 lines · Andrew Cooper
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 120
Quoted 11 lines · Peter Zijlstra
arch/x86/include/asm/debugreg.h +5
-0@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 );%dr7 has an architecturally stuck bit in it.
Quoted 4 lines
Quoted 1 line
You want *dr7 !=
0x400to avoid writing 0 unconditionally.Do we have to have that bit set when writing it? Otherwise I might actually prefer masking it out.
I'm an idiot, we write a plain 9..
Quoted 5 lines
Quoted 2 lines
Also, API wise, wouldn't it be nicer to write "dr7 =
local_db_save()" rather than having a void function returning a single long via pointer?Probably.. I started with
local_irq_save()and .. well, n/m. I'll change it ;-)How's this?
---
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 121
Quoted 9 lines
arch/x86/include/asm/debugreg.h +6
-0@@ -113,6 +113,36 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline unsigned long local_db_save (void ) +{ +unsigned long dr7; + +get_debugreg (&dr7,7 ); + dr7 ^=0x400 ;
Why xor? This seems extra confusing.
Quoted 62 lines
arch/x86/include/asm/debugreg.h +24
-0+if (dr7) +set_debugreg (0 ,7 ); +/* +* Ensure the compiler doesn't lower the above statements into +* the critical section; disabling breakpoints late would not +* be good. +*/ +barrier (); + +return dr7; +} + +static __always_inline void local_db_restore(unsigned long dr7) +{ +/* +* Ensure the compiler doesn't raise this statement into +* the critical section; enabling breakpoints early would +* not be good. +*/ +barrier (); +if (dr7) +set_debugreg (dr7,7 ); +} +#ifdef CONFIG_CPU_SUP_AMD extern void set_dr_addr_mask (unsigned long mask,int dr);#else arch/x86/kernel/traps.c +2
-16@@ -727,15 +727,7 @@ static __always_inline void debug_enter(* Entry text is excluded for HW_BP_X and cpu_entry_area, which * includes the entry stack is excluded for everything. */ -get_debugreg (*dr7,7 ); -set_debugreg (0 ,7 ); - -/* -* Ensure the compiler doesn't lower the above statements into -* the critical section; disabling breakpoints late would not -* be good. -*/ -barrier (); + *dr7 =local_db_save ();/* * The Intel SDM says: @@ -756,13 +748,7 @@ static __always_inline void debug_enter( static __always_inline void debug_exit(unsigned long dr7) { -/* -* Ensure the compiler doesn't raise this statement into -* the critical section; enabling breakpoints early would -* not be good. -*/ -barrier (); -set_debugreg (dr7,7 ); +local_db_restore (dr7); }/*
Peter Zijlstra
maintainerRe: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
Quoted 8 lines · Andy Lutomirski
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 121
Quoted 6 lines
arch/x86/include/asm/debugreg.h +6
-0+static __always_inline unsigned long local_db_save (void ) +{ +unsigned long dr7; + +get_debugreg (&dr7,7 ); + dr7 ^=0x400 ;Why xor? This seems extra confusing.
I'll do the normal mask thing ..
Re: [PATCH 1/6] x86/entry: Introduce local_db_{save,restore}()
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
Quoted 14 lines · Andrew Cooper
arch/x86/include/asm/debugreg.h static inline void debug_stack_usage_inc · line 120
Quoted 11 lines · Peter Zijlstra
arch/x86/include/asm/debugreg.h +5
-0@@ -113,6 +113,31 @@ static inline void debug_stack_usage_incstatic inline void debug_stack_usage_dec (void ) { }#endif /* X86_64 */ +static __always_inline void local_db_save(unsigned long *dr7) +{ +get_debugreg (*dr7,7 ); +if (*dr7) +set_debugreg (0 ,7 );%dr7 has an architecturally stuck bit in it.
Quoted 3 lines
Quoted 1 line
You want *dr7 !=
0x400to avoid writing 0 unconditionally.Do we have to have that bit set when writing it? Otherwise I might actually prefer masking it out.
Not currently. I guess it depends on how likely %dr7 is to gain an inverted polarity bit like %dr6 did.
~Andrew
Peter Zijlstra
maintainerPATCH 2/6 x86/entry, nmi: Disable #DB
In reply to Peter ZijlstraInstead of playing stupid games with IST stacks, fully disallow #DB during NMIs. There is absolutely no reason to allow them, and killing this saves a heap of trouble.
We already disallow #DB on noinstr and CEA, so we can't get #DB before this, and this ensures we can't get it after this either.
--- arch/x86/kernel/nmi.c | 55 ++------------------------------------------------ 1 file changed, 3 insertions(+), 52 deletions(-)
arch/x86/kernel/nmi.c +3 -52
@@ -478,40 +478,7 @@ enum nmi_states { }; static DEFINE_PER_CPU(enum nmi_states, nmi_state); static DEFINE_PER_CPU(unsigned long, nmi_cr2); - -#ifdef CONFIG_X86_64 -/* -* In x86_64, we need to handle breakpoint -> NMI -> breakpoint. Without -* some care, the inner breakpoint will clobber the outer breakpoint's -* stack. -* -* If a breakpoint is being processed, and the debug stack is being -* used, if an NMI comes in and also hits a breakpoint, the stack -* pointer will be set to the same fixed address as the breakpoint that -* was interrupted, causing that stack to be corrupted. To handle this -* case, check if the stack that was interrupted is the debug stack, and -* if so, change the IDT so that new breakpoints will use the current -* stack and not switch to the fixed address. On return of the NMI, -* switch back to the original IDT. -*/ -static DEFINE_PER_CPU(int, update_debug_stack); - -static noinstr bool is_debug_stack(unsigned long addr) -{ -struct cea_exception_stacks *cs =__this_cpu_read (cea_exception_stacks); -unsigned long top =CEA_ESTACK_TOP (cs,DB ); -unsigned long bot =CEA_ESTACK_BOT (cs,DB1 ); - -if (__this_cpu_read (debug_stack_usage)) -return true; -/* -* Note, this covers the guard page between DB and DB1 as well to -* avoid two checks. But by all means @addr can never point into -* the guard page. -*/ -return addr >= bot && addr < top; -} -#endif +static DEFINE_PER_CPU(unsigned long, nmi_dr7);DEFINE_IDTENTRY_NMI (exc_nmi ) {
@@ -526,18 +493,7 @@ DEFINE_IDTENTRY_NMI(exc_nmi)this_cpu_write (nmi_cr2,read_cr2 ()); nmi_restart: -#ifdef CONFIG_X86_64 -/* -* If we interrupted a breakpoint, it is possible that -* the nmi handler will have breakpoints too. We need to -* change the IDT such that breakpoints that happen here -* continue to use the NMI stack. -*/ -if (unlikely (is_debug_stack (regs->sp))) { -debug_stack_set_zero (); -this_cpu_write (update_debug_stack,1 ); - } -#endif +local_db_save (this_cpu_ptr (&nmi_dr7));nmi_enter ();
@@ -548,12 +504,7 @@ DEFINE_IDTENTRY_NMI(exc_nmi)nmi_exit (); -#ifdef CONFIG_X86_64 -if (unlikely (this_cpu_read (update_debug_stack))) { -debug_stack_reset (); -this_cpu_write (update_debug_stack,0 ); - } -#endif +local_db_restore (*this_cpu_ptr (&nmi_dr7));if (unlikely (this_cpu_read (nmi_cr2) !=read_cr2 ()))write_cr2 (this_cpu_read (nmi_cr2));
Peter Zijlstra
maintainerPATCH 3/6 x86/entry, mce: Disallow #DB during #MC
In reply to Peter Zijlstra#MC is fragile as heck, don't tempt fate.
--- arch/x86/kernel/cpu/mce/core.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/x86/kernel/cpu/mce/core.c +12 -0
@@ -1943,22 +1943,34 @@ static __always_inline void exc_machine_/* MCE hit kernel mode */ DEFINE_IDTENTRY_MCE (exc_machine_check) { +unsigned long dr7; + +local_db_save (&dr7);exc_machine_check_kernel (regs); +local_db_restore (dr7); }/* The user mode variant. */ DEFINE_IDTENTRY_MCE_USER (exc_machine_check) { +unsigned long dr7; + +local_db_save (&dr7);exc_machine_check_user (regs); +local_db_restore (dr7); }#else /* 32bit unified entry point */ DEFINE_IDTENTRY_MCE (exc_machine_check) { +unsigned long dr7; + +local_db_save (&dr7);if (user_mode (regs))exc_machine_check_user (regs);else exc_machine_check_kernel (regs); +local_db_restore (dr7); }#endif
Peter Zijlstra
maintainerPATCH 4/6 x86/entry: Optimize local_db_save() for virt
In reply to Peter ZijlstraBecause DRn access is 'difficult' with virt; but the DR7 read is cheaper than a cacheline miss on native, add a virt specific fast path to local_db_save(), such that when breakpoints are not in use we avoid touching DRn entirely.
--- arch/x86/include/asm/debugreg.h | 6 ++++++ arch/x86/kernel/hw_breakpoint.c | 26 ++++++++++++++++++++++---- arch/x86/kvm/vmx/nested.c | 2 ++ 3 files changed, 30 insertions(+), 4 deletions(-)
arch/x86/include/asm/debugreg.h +6 -0
@@ -115,6 +115,12 @@ static inline void debug_stack_usage_dec static __always_inline void local_db_save(unsigned long *dr7) { +if (static_cpu_has (X86_FEATURE_HYPERVISOR ) && !hw_breakpoint_active ()) { + *dr7 =0 ; +barrier (); +return ; + } +get_debugreg (*dr7,7 );if (*dr7)set_debugreg (0 ,7 );
arch/x86/kernel/hw_breakpoint.c +22 -4
@@ -97,6 +97,8 @@ int arch_install_hw_breakpoint(struct peunsigned long *dr7;int i; +lockdep_assert_irqs_disabled (); +for (i =0 ; i <HBP_NUM ; i++) {struct perf_event **slot =this_cpu_ptr (&bp_per_reg[i]);
@@ -115,6 +117,12 @@ int arch_install_hw_breakpoint(struct pe dr7 =this_cpu_ptr (&cpu_dr7); *dr7 |=encode_dr7 (i, info->len, info->type); +/* +* Ensure we first write cpu_dr7 before we set the DR7 register. +* This ensures an NMI never see cpu_dr7 0 when DR7 is not. +*/ +barrier (); +set_debugreg (*dr7,7 );if (info->mask)set_dr_addr_mask (info->mask, i);
@@ -134,9 +142,11 @@ int arch_install_hw_breakpoint(struct pevoid arch_uninstall_hw_breakpoint (struct perf_event *bp) {struct arch_hw_breakpoint *info =counter_arch_bp (bp); -unsigned long *dr7; +unsigned long dr7;int i; +lockdep_assert_irqs_disabled (); +for (i =0 ; i <HBP_NUM ; i++) {struct perf_event **slot =this_cpu_ptr (&bp_per_reg[i]);
@@ -149,12 +159,20 @@ void arch_uninstall_hw_breakpoint(structif (WARN_ONCE (i ==HBP_NUM ,"Can't find any breakpoint slot" ))return ; - dr7 =this_cpu_ptr (&cpu_dr7); - *dr7 &= ~__encode_dr7 (i, info->len, info->type); + dr7 =this_cpu_read (cpu_dr7); + dr7 &= ~__encode_dr7 (i, info->len, info->type); -set_debugreg (*dr7,7 ); +set_debugreg (dr7,7 );if (info->mask)set_dr_addr_mask (0 , i); + +/* +* Ensure the write to cpu_dr7 is after we've set the DR7 register. +* This ensures an NMI never see cpu_dr7 0 when DR7 is not. +*/ +barrier (); + +this_cpu_write (cpu_dr7, dr7); }static int arch_bp_generic_len (int x86_len)
arch/x86/kvm/vmx/nested.c +2 -0
@@ -3027,6 +3027,8 @@ static int nested_vmx_check_vmentry_hw(s/* * VMExit clears RFLAGS.IF and DR7, even on a consistency check. +* XXX how is this not broken? access to cpu_dr7 ought to be with +* IRQs disabled. */ local_irq_enable ();if (hw_breakpoint_active ())
Re: [PATCH 4/6] x86/entry: Optimize local_db_save() for virt
arch/x86/kvm/vmx/nested.c static int nested_vmx_check_vmentry_hw(s · line 3031
Quoted 9 lines · Peter Zijlstra
static int
arch_bp_generic_len(int x86_len)arch/x86/kvm/vmx/nested.c +2
-0@@ -3027,6 +3027,8 @@ static int nested_vmx_check_vmentry_hw(s/* * VMExit clears RFLAGS.IF and DR7, even on a consistency check. +* XXX how is this not broken? access to cpu_dr7 ought to be with +* IRQs disabled.
Ah, it's simply broken. This code is conditional on a module param that's off by default, i.e. it's not run widely, and odds are intersection with debugging is rare.
Moving local_irq_enable() below the DR7 restoration is not an issue.
Maybe also add lockdep_assert_irqs_disabled() to hw_breakpoint_restore() or hw_breakpoint_active()?
Quoted 5 lines
arch/x86/kvm/vmx/nested.c +0
-0*/ local_irq_enable ();if (hw_breakpoint_active ())
Peter Zijlstra
maintainerPATCH 5/6 x86/entry: Remove debug IDT frobbing
In reply to Peter ZijlstraThis is all unused now.
--- arch/x86/include/asm/debugreg.h | 19 ------------------- arch/x86/include/asm/desc.h | 34 +--------------------------------- arch/x86/kernel/cpu/common.c | 17 ----------------- arch/x86/kernel/idt.c | 30 ------------------------------ arch/x86/kernel/traps.c | 9 --------- 5 files changed, 1 insertion(+), 108 deletions(-)
arch/x86/include/asm/debugreg.h +0 -19
@@ -94,25 +94,6 @@ extern void aout_dump_debugregs(struct uextern void hw_breakpoint_restore (void ); -#ifdef CONFIG_X86_64 -DECLARE_PER_CPU (int, debug_stack_usage); -static inline void debug_stack_usage_inc (void ) -{ -__this_cpu_inc (debug_stack_usage); -} -static inline void debug_stack_usage_dec (void ) -{ -__this_cpu_dec (debug_stack_usage); -} -void debug_stack_set_zero (void ); -void debug_stack_reset (void ); -#else /* !X86_64 */ -static inline void debug_stack_set_zero (void ) { } -static inline void debug_stack_reset (void ) { } -static inline void debug_stack_usage_inc (void ) { } -static inline void debug_stack_usage_dec (void ) { } -#endif /* X86_64 */ - static __always_inline void local_db_save(unsigned long *dr7) {if (static_cpu_has (X86_FEATURE_HYPERVISOR ) && !hw_breakpoint_active ()) {
arch/x86/include/asm/desc.h +1 -33
@@ -42,8 +42,6 @@ static inline void fill_ldt(struct desc_extern struct desc_ptr idt_descr;extern gate_desc idt_table[]; -extern const struct desc_ptr debug_idt_descr; -extern gate_desc debug_idt_table[];struct gdt_page {struct desc_struct gdt[GDT_ENTRIES ];
@@ -390,31 +388,6 @@ void alloc_intr_gate(unsigned int n, conextern unsigned long system_vectors[]; -#ifdef CONFIG_X86_64 -DECLARE_PER_CPU (u32, debug_idt_ctr); -static __always_inline bool is_debug_idt_enabled(void) -{ -if (this_cpu_read (debug_idt_ctr)) -return true; - -return false; -} - -static __always_inline void load_debug_idt(void) -{ -load_idt ((const struct desc_ptr *)&debug_idt_descr); -} -#else -static inline bool is_debug_idt_enabled (void ) -{ -return false; -} - -static inline void load_debug_idt (void ) -{ -} -#endif -/* * The load_current_idt() must be called with interrupts disabled * to avoid races. That way the IDT will always be set back to the expected
@@ -424,10 +397,7 @@ static inline void load_debug_idt(void)*/ static __always_inline void load_current_idt(void) { -if (is_debug_idt_enabled ()) -load_debug_idt (); -else -load_idt ((const struct desc_ptr *)&idt_descr); +load_idt ((const struct desc_ptr *)&idt_descr); }extern void idt_setup_early_handler (void );
@@ -438,11 +408,9 @@ extern void idt_setup_apic_and_irq_gates#ifdef CONFIG_X86_64 extern void idt_setup_early_pf (void );extern void idt_setup_ist_traps (void ); -extern void idt_setup_debugidt_traps (void );#else static inline void idt_setup_early_pf (void ) { }static inline void idt_setup_ist_traps (void ) { } -static inline void idt_setup_debugidt_traps (void ) { }#endif extern void idt_invalidate (void *addr);
arch/x86/kernel/cpu/common.c +0 -17
@@ -1672,23 +1672,6 @@ void syscall_init(void) X86_EFLAGS_IOPL|X86_EFLAGS_AC|X86_EFLAGS_NT); } -DEFINE_PER_CPU (int, debug_stack_usage); -DEFINE_PER_CPU (u32, debug_idt_ctr); - -noinstr void debug_stack_set_zero(void) -{ -this_cpu_inc (debug_idt_ctr); -load_current_idt (); -} - -noinstr void debug_stack_reset(void) -{ -if (WARN_ON (!this_cpu_read (debug_idt_ctr))) -return ; -if (this_cpu_dec_return (debug_idt_ctr) ==0 ) -load_current_idt (); -} -#else /* CONFIG_X86_64 */ DEFINE_PER_CPU(struct task_struct *, current_task) = &init_task;
arch/x86/kernel/idt.c +0 -30
@@ -158,14 +158,6 @@ static const __initconst struct idt_datastatic const __initconst struct idt_data early_pf_idts[] = {INTG (X86_TRAP_PF , asm_exc_page_fault), }; - -/* -* Override for the debug_idt. Same as the default, but with interrupt -* stack set to DEFAULT_STACK (0). Required for NMI trap handling. -*/ -static const __initconst struct idt_data dbg_idts[] = { -INTG (X86_TRAP_DB , asm_exc_debug), -};#endif /* Must be page-aligned because the real IDT is used in a fixmap. */
@@ -177,9 +169,6 @@ struct desc_ptr idt_descr __ro_after_ini };#ifdef CONFIG_X86_64 -/* No need to be aligned, but done to keep all IDTs defined the same way. */ -gate_desc debug_idt_table[IDT_ENTRIES] __page_aligned_bss; -/* * The exceptions which use Interrupt stacks. They are setup after * cpu_init() when the TSS has been initialized.
@@ -192,15 +181,6 @@ static const __initconst struct idt_data ISTG(X86_TRAP_MC, asm_exc_machine_check, IST_INDEX_MCE), #endif }; - -/* -* Override for the debug_idt. Same as the default, but with interrupt -* stack set to DEFAULT_STACK (0). Required for NMI trap handling. -*/ -const struct desc_ptr debug_idt_descr = { - .size =IDT_ENTRIES *16 -1 , - .address = (unsigned long ) debug_idt_table, -};#endif static inline void idt_init_desc (gate_desc *gate,const struct idt_data *d)
@@ -292,16 +272,6 @@ void __init idt_setup_ist_traps(void) {idt_setup_from_table (idt_table, ist_idts,ARRAY_SIZE (ist_idts), true); } - -/** -* idt_setup_debugidt_traps - Initialize the debug idt table with debug traps -*/ -void __init idt_setup_debugidt_traps (void ) -{ -memcpy (&debug_idt_table, &idt_table,IDT_ENTRIES *16 ); - -idt_setup_from_table (debug_idt_table, dbg_idts,ARRAY_SIZE (dbg_idts), false); -}#endif /**
arch/x86/kernel/traps.c +0 -9
@@ -798,12 +798,6 @@ static void noinstr handle_debug(structreturn ; } -/* -* Let others (NMI) know that the debug stack is in use -* as we may switch to the interrupt stack. -*/ -debug_stack_usage_inc (); -/* It's safe to allow irq's after DR6 has been saved */ cond_local_irq_enable (regs);
@@ -831,7 +825,6 @@ static void noinstr handle_debug(struct out:cond_local_irq_disable (regs); -debug_stack_usage_dec ();instrumentation_end (); }
@@ -1077,6 +1070,4 @@ void __init trap_init(void)cpu_init ();idt_setup_ist_traps (); - -idt_setup_debugidt_traps (); }
Peter Zijlstra
maintainerRFC PATCH 6/6 x86/entry: Remove DBn stacks
In reply to Peter ZijlstraBoth #DB itself, as all other IST users (NMI, #MC) now clear DR7 on entry. Combined with not allowing breakpoints on entry/noinstr/NOKPROBE text and no single step (EFLAGS.TF) inside the #DB handler should guarantee us no nested #DB.
XXX depends on KGDB breakpoints not being stupid
--- arch/x86/entry/entry_64.S | 13 ------------- arch/x86/include/asm/cpu_entry_area.h | 6 ------ arch/x86/kernel/asm-offsets_64.c | 3 --- arch/x86/kernel/dumpstack_64.c | 3 --- arch/x86/mm/cpu_entry_area.c | 1 - 5 files changed, 26 deletions(-)
arch/x86/entry/entry_64.S +0 -13
@@ -396,11 +396,6 @@ SYM_CODE_END(\asmsym) idtentry \vector asm_\cfunc \cfunc has_error_code=0 .endm -/* - * MCE and DB exceptions - */ -#define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8) - /** * idtentry_mce_db - Macro to generate entry stubs for #MC and #DB * @vector: Vector number
@@ -445,16 +440,8 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ - .if \vector == X86_TRAP_DB - subq $DB_STACK_OFFSET, CPU_TSS_IST(IST_INDEX_DB) - .endif - call \cfunc - .if \vector == X86_TRAP_DB - addq $DB_STACK_OFFSET, CPU_TSS_IST(IST_INDEX_DB) - .endif - jmp paranoid_exit /* Switch to the regular task stack and use the noist entry point */
arch/x86/include/asm/cpu_entry_area.h +0 -6
@@ -16,10 +16,6 @@char DF_stack[EXCEPTION_STKSZ ]; \char NMI_stack_guard[guardsize]; \char NMI_stack[EXCEPTION_STKSZ ]; \ -char DB2_stack_guard[guardsize]; \ -char DB2_stack[db2_holesize]; \ -char DB1_stack_guard[guardsize]; \ -char DB1_stack[EXCEPTION_STKSZ ]; \char DB_stack_guard[guardsize]; \char DB_stack[EXCEPTION_STKSZ ]; \char MCE_stack_guard[guardsize]; \
@@ -42,8 +38,6 @@ struct cea_exception_stacks {enum exception_stack_ordering {ESTACK_DF ,ESTACK_NMI , -ESTACK_DB2 , -ESTACK_DB1 ,ESTACK_DB ,ESTACK_MCE ,N_EXCEPTION_STACKS
arch/x86/kernel/asm-offsets_64.c +0 -3
@@ -57,9 +57,6 @@ int main(void)BLANK ();#undef ENTRY -OFFSET (TSS_ist, tss_struct, x86_tss.ist); -DEFINE (DB_STACK_OFFSET , offsetof(struct cea_exception_stacks , DB_stack) - - offsetof(struct cea_exception_stacks , DB1_stack));BLANK ();#ifdef CONFIG_STACKPROTECTOR
arch/x86/kernel/dumpstack_64.c +0 -3
@@ -22,8 +22,6 @@static const char *const exception_stack_names[] = { [ESTACK_DF ] ="#DF" , [ESTACK_NMI ] ="NMI" , - [ESTACK_DB2 ] ="#DB2" , - [ESTACK_DB1 ] ="#DB1" , [ESTACK_DB ] ="#DB" , [ESTACK_MCE ] ="#MC" , };
@@ -79,7 +77,6 @@ static conststruct estack_pages estack_pages[CEA_ESTACK_PAGES ]____cacheline_aligned = {EPAGERANGE (DF ),EPAGERANGE (NMI ), -EPAGERANGE (DB1 ),EPAGERANGE (DB ),EPAGERANGE (MCE ), };
arch/x86/mm/cpu_entry_area.c +0 -1
@@ -107,7 +107,6 @@ static void __init percpu_setup_exceptio*/ cea_map_stack (DF );cea_map_stack (NMI ); -cea_map_stack (DB1 );cea_map_stack (DB );cea_map_stack (MCE ); }
Re: [RFC][PATCH 6/6] x86/entry: Remove DBn stacks
arch/x86/entry/entry_64.S SYM_CODE_START(\asmsym)
Quoted 32 lines · Peter Zijlstra
Both #DB itself, as all other IST users (NMI, #MC) now clear DR7 on entry. Combined with not allowing breakpoints on entry/noinstr/NOKPROBE text and no single step (EFLAGS.TF) inside the #DB handler should guarantee us no nested #DB.
XXX depends on KGDB breakpoints not being stupid
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/entry/entry_64.S | 13 ------------- arch/x86/include/asm/cpu_entry_area.h | 6 ------ arch/x86/kernel/asm-offsets_64.c | 3 --- arch/x86/kernel/dumpstack_64.c | 3 --- arch/x86/mm/cpu_entry_area.c | 1 - 5 files changed, 26 deletions(-)arch/x86/entry/entry_64.S +0
-5@@ -396,11 +396,6 @@ SYM_CODE_END(\asmsym) idtentry \vector asm_\cfunc \cfunc has_error_code=0 .endm -/* - * MCE and DB exceptions - */ -#define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8) - /** * idtentry_mce_db - Macro to generate entry stubs for #MC and #DB * @vector: Vector number@@ -445,16 +440,8 @@ SYM_CODE_START(\asmsym)
There is some comment in the code above which need to be removed:
* If the trap is #DB then the interrupt stack entry in the IST is
* moved to the second stack, so a potential recursion will have a
* fresh IST.
Quoted 19 lines
arch/x86/entry/entry_64.S +0
-8movq %rsp, %rdi /* pt_regs pointer */ - .if \vector == X86_TRAP_DB - subq $DB_STACK_OFFSET, CPU_TSS_IST(IST_INDEX_DB) - .endif - call \cfunc - .if \vector == X86_TRAP_DB - addq $DB_STACK_OFFSET, CPU_TSS_IST(IST_INDEX_DB) - .endif - jmp paranoid_exit /* Switch to the regular task stack and use the noist entry point */
arch/x86/include/asm/cpu_entry_area.h line 21
Quoted 10 lines
arch/x86/include/asm/cpu_entry_area.h +0
-4@@ -16,10 +16,6 @@char DF_stack[EXCEPTION_STKSZ ]; \char NMI_stack_guard[guardsize]; \char NMI_stack[EXCEPTION_STKSZ ]; \ -char DB2_stack_guard[guardsize]; \ -char DB2_stack[db2_holesize]; \ -char DB1_stack_guard[guardsize]; \ -char DB1_stack[EXCEPTION_STKSZ ]; \char DB_stack_guard[guardsize]; \char DB_stack[EXCEPTION_STKSZ ]; \char MCE_stack_guard[guardsize]; \
argument db2_holesize should be removed
arch/x86/include/asm/cpu_entry_area.h struct cea_exception_stacks { · line 43
Quoted 9 lines
arch/x86/include/asm/cpu_entry_area.h +0
-2@@ -42,8 +38,6 @@ struct cea_exception_stacks {enum exception_stack_ordering {ESTACK_DF ,ESTACK_NMI , -ESTACK_DB2 , -ESTACK_DB1 ,ESTACK_DB ,ESTACK_MCE ,N_EXCEPTION_STACKS
There will be a compile error about N_EXCEPTION_STACKS in arch/x86/kernel/dumpstack_64.c. It should have
- BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
+ BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
Quoted 43 lines
arch/x86/kernel/asm-offsets_64.c +0
-3@@ -57,9 +57,6 @@ int main(void)BLANK ();#undef ENTRY -OFFSET (TSS_ist, tss_struct, x86_tss.ist); -DEFINE (DB_STACK_OFFSET , offsetof(struct cea_exception_stacks , DB_stack) - - offsetof(struct cea_exception_stacks , DB1_stack));BLANK ();#ifdef CONFIG_STACKPROTECTOR arch/x86/kernel/dumpstack_64.c +0
-3@@ -22,8 +22,6 @@static const char *const exception_stack_names[] = { [ESTACK_DF ] ="#DF" , [ESTACK_NMI ] ="NMI" , - [ESTACK_DB2 ] ="#DB2" , - [ESTACK_DB1 ] ="#DB1" , [ESTACK_DB ] ="#DB" , [ESTACK_MCE ] ="#MC" , };@@ -79,7 +77,6 @@ static conststruct estack_pages estack_pages[CEA_ESTACK_PAGES ]____cacheline_aligned = {EPAGERANGE (DF ),EPAGERANGE (NMI ), -EPAGERANGE (DB1 ),EPAGERANGE (DB ),EPAGERANGE (MCE ), };arch/x86/mm/cpu_entry_area.c +0
-1@@ -107,7 +107,6 @@ static void __init percpu_setup_exceptio*/ cea_map_stack (DF );cea_map_stack (NMI ); -cea_map_stack (DB1 );cea_map_stack (DB );cea_map_stack (MCE ); }
Peter Zijlstra
maintainerRe: [RFC][PATCH 6/6] x86/entry: Remove DBn stacks
Quoted 5 lines · Lai Jiangshan
There will be a compile error about
N_EXCEPTION_STACKSinarch/x86/kernel/dumpstack_64.c. It should have-
BUILD_BUG_ON(N_EXCEPTION_STACKS != 6);
+BUILD_BUG_ON(N_EXCEPTION_STACKS != 4);
Yeah, I actually have that, but forgot to refresh before sending :/
Quoted 3 lines · Peter Zijlstra
These patches disallow #DB during NMI/#MC and allow removing a lot of fugly code.
Hello
Will #DB be allowed in #DF?
Thanks
Lai
Quoted 4 lines
Patch #6 should probably wait until we've got the KGDB situation sorted because applying that makes a bad situation worse.
Peter Zijlstra
maintainerQuoted 8 lines · Lai Jiangshan
Quoted 3 lines · Peter Zijlstra
These patches disallow #DB during NMI/#MC and allow removing a lot of fugly code.
Hello
Will #DB be allowed in #DF?
No, that whole thing is noinstr.
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Lai Jiangshan
Quoted 1 line · Peter Zijlstra
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
These patches disallow #DB during NMI/#MC and allow removing a lot of fugly code.
Hello
Will #DB be allowed in #DF?
No, that whole thing is noinstr.
But it calls many functions, including die(), panic(). We don't want #DB to interfere how it die() and panic(). Since it is in fragile #DF, the #DB may mess it up and make #DF fails to report and die.
Peter Zijlstra
maintainerQuoted 10 lines · Lai Jiangshan
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Lai Jiangshan
Quoted 3 lines · Peter Zijlstra
These patches disallow #DB during NMI/#MC and allow removing a lot of fugly code.
Hello
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Will #DB be allowed in #DF?
No, that whole thing is noinstr.
But it calls many functions, including
die(),panic(). We don't want #DB to interfere how itdie()andpanic(). Since it is in fragile #DF, the #DB may mess it up and make #DF fails to report and die.
The only recoverable #DF is the ESPFIX shit. If we do not take that, we're on the way to panic(), I really can't be arsed if you crash the box before that, we're going to die anyway.