perf_counter: x86: Fix call-chain support to use NMI-safe methods
Subsystems (3)
Commit-ID: 74193ef0ecab92535c8517f082f1f50504526c9b Gitweb: http://git.kernel.org/tip/74193ef0ecab92535c8517f082f1f50504526c9b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 15 Jun 2009 13:07:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Jun 2009 15:57:53 +0200
perf_counter: x86: Fix call-chain support to use NMI-safe methods
__copy_from_user_inatomic() isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.
Therefore use a GUP based approach to copy the stack frames over.
We tried an alternative solution as well: we used a forward ported version of Mathieu Desnoyers's "NMI safe INT3 and Page Fault" patch that modifies the exception return path to use an open-coded IRET with explicit stack unrolling and TF checking.
This didnt work as it interacted with faulting user-space instructions, causing them not to restart properly, which corrupts user-space registers.
Solving that would probably involve disassembling those instructions and backtracing the RIP. But even without that, the code was deemed rather complex to the already non-trivial x86 entry assembly code, so instead we went for this GUP based method that does a software-walk of the pagetables.
LKML-Reference: <new-submission>
--- arch/x86/kernel/cpu/perf_counter.c | 49 ++++++++++++++++++++++++++++------- 1 files changed, 39 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/perf_counter.c +39 -10
@@ -19,6 +19,7 @@#include <linux/kdebug.h> #include <linux/sched.h> #include <linux/uaccess.h> +#include <linux/highmem.h> #include <asm/apic.h> #include <asm/stacktrace.h>
@@ -1617,20 +1618,48 @@ perf_callchain_kernel(struct pt_regs *regs, struct perf_callchain_entry *entry) entry->kernel = entry->nr - nr; } -static int copy_stack_frame (const void __user *fp,struct stack_frame *frame) +/* +* best effort, GUP based copy_from_user() that assumes IRQ or NMI context +*/ +static unsigned long +copy_from_user_nmi (void *to,const void __user *from,unsigned long n) { +unsigned long offset, addr = (unsigned long )from; +int type =in_nmi () ?KM_NMI :KM_IRQ0 ; +unsigned long size, len =0 ; +struct page *page; +void *map;int ret; -if (!access_ok (VERIFY_READ , fp,sizeof (*frame))) -return 0 ; +do { + ret =__get_user_pages_fast (addr,1 ,0 , &page); +if (!ret) +break ; - ret =1 ; -pagefault_disable (); -if (__copy_from_user_inatomic (frame, fp,sizeof (*frame))) - ret =0 ; -pagefault_enable (); + offset = addr & (PAGE_SIZE -1 ); + size =min (PAGE_SIZE - offset, n - len); -return ret; + map =kmap_atomic (page, type); +memcpy (to, map+offset, size); +kunmap_atomic (map, type); +put_page (page); + + len += size; + to += size; + addr += size; + + }while (len < n); + +return len; +} + +static int copy_stack_frame (const void __user *fp,struct stack_frame *frame) +{ +unsigned long bytes; + + bytes =copy_from_user_nmi (frame, fp,sizeof (*frame)); + +return bytes ==sizeof (*frame); } static void
@@ -1643,7 +1672,7 @@ perf_callchain_user(struct pt_regs *regs, struct perf_callchain_entry *entry)if (!user_mode (regs)) regs =task_pt_regs (current); - fp = (void __user *)regs->bp; + fp = (void __user *)regs->bp;callchain_store (entry, regs->ip);
Quoted 24 lines · * tip-bot for Peter Zijlstra ([email protected]) wrote:
Commit-ID: 74193ef0ecab92535c8517f082f1f50504526c9b Gitweb: http://git.kernel.org/tip/74193ef0ecab92535c8517f082f1f50504526c9b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 15 Jun 2009 13:07:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Jun 2009 15:57:53 +0200
perf_counter: x86: Fix call-chain support to use NMI-safe methods
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.Therefore use a GUP based approach to copy the stack frames over.
We tried an alternative solution as well: we used a forward ported version of Mathieu Desnoyers's "NMI safe INT3 and Page Fault" patch that modifies the exception return path to use an open-coded IRET with explicit stack unrolling and TF checking.
This didnt work as it interacted with faulting user-space instructions, causing them not to restart properly, which corrupts user-space registers.
Quoted 6 lines
Solving that would probably involve disassembling those instructions and backtracing the RIP. But even without that, the code was deemed rather complex to the already non-trivial x86 entry assembly code, so instead we went for this GUP based method that does a software-walk of the pagetables.
Hrm, I'm probably missing something. Normally, you should test for "in_nmi()" upon return from exception, and only in these cases go for the open-coded IRET with stack unrolling and ret. I really don't see how you end up messing up the page fault return to userspace path, as it's impossible to have in_nmi() set.
Mathieu
Quoted 97 lines
Signed-off-by Peter Zijlstra <[email protected]>Cc Nick Piggin <[email protected]>Cc Pekka Enberg <[email protected]>Cc Vegard Nossum <[email protected]>Cc Jeremy Fitzhardinge <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Linus Torvalds <[email protected]>Cc Mike Galbraith <[email protected]>Cc Paul Mackerras <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>LKML-Reference: <new-submission>
Signed-off-by Ingo Molnar <[email protected]>--- arch/x86/kernel/cpu/perf_counter.c | 49 ++++++++++++++++++++++++++++------- 1 files changed, 39 insertions(+), 10 deletions(-)arch/x86/kernel/cpu/perf_counter.c +39
-10@@ -19,6 +19,7 @@#include <linux/kdebug.h> #include <linux/sched.h> #include <linux/uaccess.h> +#include <linux/highmem.h> #include <asm/apic.h> #include <asm/stacktrace.h> @@ -1617,20 +1618,48 @@ perf_callchain_kernel(struct pt_regs *regs, struct perf_callchain_entry *entry) entry->kernel = entry->nr - nr; } -static int copy_stack_frame (const void __user *fp,struct stack_frame *frame) +/* +* best effort, GUP based copy_from_user() that assumes IRQ or NMI context +*/ +static unsigned long +copy_from_user_nmi (void *to,const void __user *from,unsigned long n) { +unsigned long offset, addr = (unsigned long )from; +int type =in_nmi () ?KM_NMI :KM_IRQ0 ; +unsigned long size, len =0 ; +struct page *page; +void *map;int ret; -if (!access_ok (VERIFY_READ , fp,sizeof (*frame))) -return 0 ; +do { + ret =__get_user_pages_fast (addr,1 ,0 , &page); +if (!ret) +break ; - ret =1 ; -pagefault_disable (); -if (__copy_from_user_inatomic (frame, fp,sizeof (*frame))) - ret =0 ; -pagefault_enable (); + offset = addr & (PAGE_SIZE -1 ); + size =min (PAGE_SIZE - offset, n - len); -return ret; + map =kmap_atomic (page, type); +memcpy (to, map+offset, size); +kunmap_atomic (map, type); +put_page (page); + + len += size; + to += size; + addr += size; + + }while (len < n); + +return len; +} + +static int copy_stack_frame (const void __user *fp,struct stack_frame *frame) +{ +unsigned long bytes; + + bytes =copy_from_user_nmi (frame, fp,sizeof (*frame)); + +return bytes ==sizeof (*frame); } static void@@ -1643,7 +1672,7 @@ perf_callchain_user(struct pt_regs *regs, struct perf_callchain_entry *entry)if (!user_mode (regs)) regs =task_pt_regs (current); - fp = (void __user *)regs->bp; + fp = (void __user *)regs->bp;callchain_store (entry, regs->ip);
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 32 lines · Mathieu Desnoyers
Quoted 24 lines · * tip-bot for Peter Zijlstra ([email protected]) wrote:
Commit-ID: 74193ef0ecab92535c8517f082f1f50504526c9b Gitweb: http://git.kernel.org/tip/74193ef0ecab92535c8517f082f1f50504526c9b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 15 Jun 2009 13:07:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Jun 2009 15:57:53 +0200
perf_counter: x86: Fix call-chain support to use NMI-safe methods
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.Therefore use a GUP based approach to copy the stack frames over.
We tried an alternative solution as well: we used a forward ported version of Mathieu Desnoyers's "NMI safe INT3 and Page Fault" patch that modifies the exception return path to use an open-coded IRET with explicit stack unrolling and TF checking.
This didnt work as it interacted with faulting user-space instructions, causing them not to restart properly, which corrupts user-space registers.
Quoted 6 lines
Solving that would probably involve disassembling those instructions and backtracing the RIP. But even without that, the code was deemed rather complex to the already non-trivial x86 entry assembly code, so instead we went for this GUP based method that does a software-walk of the pagetables.
Quoted 5 lines
Hrm, I'm probably missing something. Normally, you should test for "
in_nmi()" upon return from exception, and only in these cases go for the open-coded IRET with stack unrolling and ret. I really don't see how you end up messing up the page fault return to userspace path, as it's impossible to havein_nmi()set.
here's the (heavily modified) version of your patch that i used.
Ingo
-------------------->
Subject: x86 NMI-safe INT3 and Page Fault
From: Mathieu Desnoyers <[email protected]>
Date: Mon, 12 May 2008 21:21:07 +0200
Implements an alternative iret with popf and return so trap and exception handlers can return to the NMI handler without issuing iret. iret would cause NMIs to be reenabled prematurely. x86_32 uses popf and far return. x86_64 has to copy the return instruction pointer to the top of the previous stack, issue a popf, loads the previous esp and issue a near return (ret).
It allows placing immediate values (and therefore optimized trace_marks) in NMI code since returning from a breakpoint would be valid. Accessing vmalloc'd memory, which allows executing module code or accessing vmapped or vmalloc'd areas from NMI context, would also be valid. This is very useful to tracers like LTTng.
This patch makes all faults, traps and exception safe to be called from NMI context *except* single-stepping, which requires iret to restore the TF (trap flag) and jump to the return address in a single instruction. Sorry, no kprobes support in NMI handlers because of this limitation. We cannot single-step an NMI handler, because iret must set the TF flag and return back to the instruction to single-step in a single instruction. This cannot be emulated with popf/lret, because lret would be single-stepped. It does not apply to immediate values because they do not use single-stepping. This code detects if the TF flag is set and uses the iret path for single-stepping, even if it reactivates NMIs prematurely.
Test to detect if nested under a NMI handler is only done upon the return from trap/exception to kernel, which is not frequent. Other return paths (return from trap/exception to userspace, return from interrupt) keep the exact same behavior (no slowdown).
Depends on :
change-alpha-active-count-bit.patch
change-avr32-active-count-bit.patch
TODO : test with lguest, xen, kvm.
** This patch depends on the "Stringify support commas" patchset ** ** Also depends on fix-x86_64-page-fault-scheduler-race patch **
tested on x86_32 (tests implemented in a separate patch) :
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
- trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
- tested vmalloc faults in NMI handler by placing a non-optimized marker in the NMI handler (so no breakpoint is executed) and connecting a probe which touches every pages of a 20MB vmalloc'd buffer. It executes trough the return path without problem.
- Tested with and without preemption
tested on x86_64
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
- trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
To test on x86_64 :
- Test without preemption
- Test vmalloc faults
- Test on Intel 64 bits CPUs. (AMD64 was fine)
Changelog since v1 :
- x86_64 fixes.
Changelog since v2 :
- fix paravirt build
Changelog since v3 :
- Include modifications suggested by Jeremy
Changelog since v4 :
- including hardirq.h in entry_32/64.S is a bad idea (non ifndef'd C code), define HARDNMI_MASK in the .S files directly.
Changelog since v5 :
- Add HARDNMI_MASK to irq_count() and make die() more verbose for NMIs. Changelog since v7 :
- Implement paravirtualized nmi_return.
Changelog since v8 :
- refreshed the patch for asm-offsets. Those were left out of v8.
- now depends on "Stringify support commas" patch.
Changelog since v9 :
- Only test the nmi nested preempt count flag upon return from exceptions, not on return from interrupts. Only the kernel return path has this test.
- Add Xen, VMI, lguest support. Use their iret pavavirt ops in lieu of nmi_return.
-- Ported to sched-devel.git
--- arch/x86/include/asm/irqflags.h | 56 +++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/dumpstack.c | 2 + arch/x86/kernel/entry_32.S | 30 +++++++++++++++++++++ arch/x86/kernel/entry_64.S | 57 +++++++++++++++++++++++++++++++--------- include/linux/hardirq.h | 16 +++++++---- 5 files changed, 144 insertions(+), 17 deletions(-)
arch/x86/include/asm/irqflags.h +56 -0
@@ -51,6 +51,61 @@ static inline void native_halt(void)#endif +#ifdef CONFIG_X86_64 +/* +* Only returns from a trap or exception to a NMI context (intra-privilege +* level near return) to the same SS and CS segments. Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. It takes care of modifying the eflags, rsp and returning to the +* previous function. +* +* The stack, at that point, looks like : +* +* 0(rsp) RIP +* 8(rsp) CS +* 16(rsp) EFLAGS +* 24(rsp) RSP +* 32(rsp) SS +* +* Upon execution : +* Copy EIP to the top of the return stack +* Update top of return stack address +* Pop eflags into the eflags register +* Make the return stack current +* Near return (popping the return address from the return stack) +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret +#else +/* +* Protected mode only, no V8086. Implies that protected mode must +* be entered before NMIs or MCEs are enabled. Only returns from a trap or +* exception to a NMI context (intra-privilege level far return). Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. +* +* The stack, at that point, looks like : +* +* 0(esp) EIP +* 4(esp) CS +* 8(esp) EFLAGS +* +* Upon execution : +* Copy the stack eflags to top of stack +* Pop eflags into the eflags register +* Far return: pop EIP and CS into their register, and additionally pop EFLAGS. +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushl 8(%esp); \ + popfl; \ + lret $4 +#endif +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else
@@ -109,6 +164,7 @@ static inline unsigned long __raw_local_#define ENABLE_INTERRUPTS (x) sti#define DISABLE_INTERRUPTS (x) cli +#define INTERRUPT_RETURN_NMI_SAFE NATIVE_INTERRUPT_RETURN_NMI_SAFE#ifdef CONFIG_X86_64 #define SWAPGS swapgs
arch/x86/kernel/dumpstack.c +2 -0
@@ -237,6 +237,8 @@ void __kprobes oops_end(unsigned long flif (!signr)return ; +if (in_nmi ()) +panic ("Fatal exception in non-maskable interrupt" );if (in_interrupt ())panic ("Fatal exception in interrupt" );if (panic_on_oops)
arch/x86/kernel/entry_32.S +30 -0
@@ -80,6 +80,8 @@ #define nr_syscalls ((syscall_table_size)/4) +#define HARDNMI_MASK 0x40000000 + #ifdef CONFIG_PREEMPT #define preempt_stop(clobbers) DISABLE_INTERRUPTS(clobbers); TRACE_IRQS_OFF #else
@@ -344,8 +346,32 @@ END(ret_from_fork) # userspace resumption stub bypassing syscall exit tracing ALIGN RING0_PTREGS_FRAME + ret_from_exception: preempt_stop(CLBR_ANY) + GET_THREAD_INFO(%ebp) + movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS + movb PT_CS(%esp), %al + andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax + cmpl $USER_RPL, %eax + jae resume_userspace # returning to v8086 or userspace + testl $HARDNMI_MASK,TI_preempt_count(%ebp) + jz resume_kernel /* Not nested over NMI ? */ + testw $X86_EFLAGS_TF, PT_EFLAGS(%esp) + jnz resume_kernel /* + * If single-stepping an NMI handler, + * use the normal iret path instead of + * the popf/lret because lret would be + * single-stepped. It should not + * happen : it will reactivate NMIs + * prematurely. + */ + TRACE_IRQS_IRET + RESTORE_REGS + addl $4, %esp # skip orig_eax/error_code + CFI_ADJUST_CFA_OFFSET -4 + INTERRUPT_RETURN_NMI_SAFE + ret_from_intr: GET_THREAD_INFO(%ebp) check_userspace:
@@ -851,6 +877,10 @@ ENTRY(native_iret) .previous END(native_iret) +ENTRY(native_nmi_return) + NATIVE_INTERRUPT_RETURN_NMI_SAFE # Should we deal with popf exception ? +END(native_nmi_return) + ENTRY(native_irq_enable_sysexit) sti sysexit
arch/x86/kernel/entry_64.S +45 -12
104 lines not on this page · raw message
include/linux/hardirq.h +11 -5
@@ -1,12 +1,14 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#ifndef __ASSEMBLY__#include <linux/preempt.h> #include <linux/smp_lock.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h> #include <asm/hardirq.h> #include <asm/system.h> +#endif /* * We put the hardirq and softirq counter into the preemption
@@ -50,17 +52,17 @@#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) -#define __IRQ_MASK (x) ((1UL << (x))-1) +#define __IRQ_MASK (x) ((1 << (x))-1)#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) -#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT) -#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) -#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) -#define NMI_OFFSET (1UL << NMI_SHIFT) +#define PREEMPT_OFFSET (1 << PREEMPT_SHIFT) +#define SOFTIRQ_OFFSET (1 << SOFTIRQ_SHIFT) +#define HARDIRQ_OFFSET (1 << HARDIRQ_SHIFT) +#define NMI_OFFSET (1 << NMI_SHIFT)#if PREEMPT_ACTIVE < (1 << (NMI_SHIFT +NMI_BITS ))#error PREEMPT_ACTIVE is too low!
@@ -116,6 +118,8 @@# define IRQ_EXIT_OFFSET HARDIRQ_OFFSET#endif +#ifndef __ASSEMBLY__ +#if defined(CONFIG_SMP ) || defined(CONFIG_GENERIC_HARDIRQS )extern void synchronize_irq (unsigned int irq);#else
@@ -195,4 +199,6 @@ extern void irq_exit(void);ftrace_nmi_exit (); \ } while (0) +#endif /* !__ASSEMBLY__ */ +#endif /* LINUX_HARDIRQ_H */
Quoted 242 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 32 lines · Mathieu Desnoyers
Quoted 24 lines · * tip-bot for Peter Zijlstra ([email protected]) wrote:
Commit-ID: 74193ef0ecab92535c8517f082f1f50504526c9b Gitweb: http://git.kernel.org/tip/74193ef0ecab92535c8517f082f1f50504526c9b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 15 Jun 2009 13:07:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Jun 2009 15:57:53 +0200
perf_counter: x86: Fix call-chain support to use NMI-safe methods
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.Therefore use a GUP based approach to copy the stack frames over.
We tried an alternative solution as well: we used a forward ported version of Mathieu Desnoyers's "NMI safe INT3 and Page Fault" patch that modifies the exception return path to use an open-coded IRET with explicit stack unrolling and TF checking.
This didnt work as it interacted with faulting user-space instructions, causing them not to restart properly, which corrupts user-space registers.
Quoted 6 lines
Solving that would probably involve disassembling those instructions and backtracing the RIP. But even without that, the code was deemed rather complex to the already non-trivial x86 entry assembly code, so instead we went for this GUP based method that does a software-walk of the pagetables.
Quoted 5 lines
Hrm, I'm probably missing something. Normally, you should test for "
in_nmi()" upon return from exception, and only in these cases go for the open-coded IRET with stack unrolling and ret. I really don't see how you end up messing up the page fault return to userspace path, as it's impossible to havein_nmi()set.here's the (heavily modified) version of your patch that i used.
Ingo
-------------------->
Subject: x86 NMI-safe INT3 and Page Fault
From: Mathieu Desnoyers <[email protected]>
Date: Mon, 12 May 2008 21:21:07 +0200Implements an alternative iret with popf and return so trap and exception handlers can return to the NMI handler without issuing iret. iret would cause NMIs to be reenabled prematurely.
x86_32uses popf and far return.x86_64has to copy the return instruction pointer to the top of the previous stack, issue a popf, loads the previous esp and issue a near return (ret).It allows placing immediate values (and therefore optimized
trace_marks) in NMI code since returning from a breakpoint would be valid. Accessing vmalloc'd memory, which allows executing module code or accessing vmapped or vmalloc'd areas from NMI context, would also be valid. This is very useful to tracers like LTTng.This patch makes all faults, traps and exception safe to be called from NMI context *except* single-stepping, which requires iret to restore the TF (trap flag) and jump to the return address in a single instruction. Sorry, no kprobes support in NMI handlers because of this limitation. We cannot single-step an NMI handler, because iret must set the TF flag and return back to the instruction to single-step in a single instruction. This cannot be emulated with popf/lret, because lret would be single-stepped. It does not apply to immediate values because they do not use single-stepping. This code detects if the TF flag is set and uses the iret path for single-stepping, even if it reactivates NMIs prematurely.
Test to detect if nested under a NMI handler is only done upon the return from trap/exception to kernel, which is not frequent. Other return paths (return from trap/exception to userspace, return from interrupt) keep the exact same behavior (no slowdown).
Depends on :
change-alpha-active-count-bit.patch
change-avr32-active-count-bit.patchTODO : test with lguest, xen, kvm.
** This patch depends on the "Stringify support commas" patchset ** ** Also depends on fix-x86_64-page-fault-scheduler-race patch **
tested on
x86_32(tests implemented in a separate patch) :
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
- tested vmalloc faults in NMI handler by placing a non-optimized marker in the NMI handler (so no breakpoint is executed) and connecting a probe which touches every pages of a 20MB vmalloc'd buffer. It executes trough the return path without problem.
- Tested with and without preemptiontested on
x86_64
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.To test on
x86_64:
- Test without preemption
- Test vmalloc faults
- Test on Intel 64 bits CPUs. (AMD64 was fine)Changelog since v1 :
-x86_64fixes.
Changelog since v2 :
- fix paravirt build
Changelog since v3 :
- Include modifications suggested by Jeremy
Changelog since v4 :
- includinghardirq.hin entry_32/64.S is a bad idea (non ifndef'd C code), defineHARDNMI_MASKin the .S files directly.
Changelog since v5 :
- AddHARDNMI_MASKtoirq_count()and makedie()more verbose for NMIs. Changelog since v7 :
- Implement paravirtualizednmi_return.
Changelog since v8 :
- refreshed the patch for asm-offsets. Those were left out of v8.
- now depends on "Stringify support commas" patch.
Changelog since v9 :
- Only test the nmi nested preempt count flag upon return from exceptions, not on return from interrupts. Only the kernel return path has this test.
- Add Xen, VMI, lguest support. Use their iret pavavirt ops in lieu ofnmi_return.-- Ported to sched-devel.git
Signed-off-by Mathieu Desnoyers <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Jeremy Fitzhardinge <[email protected]>CC Steven Rostedt <[email protected]>CC "Frank Ch. Eigler" <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/include/asm/irqflags.h | 56 +++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/dumpstack.c | 2 + arch/x86/kernel/entry_32.S | 30 +++++++++++++++++++++ arch/x86/kernel/entry_64.S | 57 +++++++++++++++++++++++++++++++--------- include/linux/hardirq.h | 16 +++++++---- 5 files changed, 144 insertions(+), 17 deletions(-)Index:
linux/arch/x86/include/asm/irqflags.h
===================================================================arch/x86/include/asm/irqflags.h +56
-0@@ -51,6 +51,61 @@ static inline void native_halt(void)#endif +#ifdef CONFIG_X86_64 +/* +* Only returns from a trap or exception to a NMI context (intra-privilege +* level near return) to the same SS and CS segments. Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. It takes care of modifying the eflags, rsp and returning to the +* previous function. +* +* The stack, at that point, looks like : +* +* 0(rsp) RIP +* 8(rsp) CS +* 16(rsp) EFLAGS +* 24(rsp) RSP +* 32(rsp) SS +* +* Upon execution : +* Copy EIP to the top of the return stack +* Update top of return stack address +* Pop eflags into the eflags register +* Make the return stack current +* Near return (popping the return address from the return stack) +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret +#else +/* +* Protected mode only, no V8086. Implies that protected mode must +* be entered before NMIs or MCEs are enabled. Only returns from a trap or +* exception to a NMI context (intra-privilege level far return). Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. +* +* The stack, at that point, looks like : +* +* 0(esp) EIP +* 4(esp) CS +* 8(esp) EFLAGS +* +* Upon execution : +* Copy the stack eflags to top of stack +* Pop eflags into the eflags register +* Far return: pop EIP and CS into their register, and additionally pop EFLAGS. +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushl 8(%esp); \ + popfl; \ + lret $4 +#endif +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else @@ -109,6 +164,7 @@ static inline unsigned long __raw_local_#define ENABLE_INTERRUPTS (x) sti#define DISABLE_INTERRUPTS (x) cli +#define INTERRUPT_RETURN_NMI_SAFE NATIVE_INTERRUPT_RETURN_NMI_SAFE#ifdef CONFIG_X86_64 #define SWAPGS swapgsIndex:
linux/arch/x86/kernel/dumpstack.c
===================================================================arch/x86/kernel/dumpstack.c +2
-0@@ -237,6 +237,8 @@ void __kprobes oops_end(unsigned long flif (!signr)return ; +if (in_nmi ()) +panic ("Fatal exception in non-maskable interrupt" );if (in_interrupt ())panic ("Fatal exception in interrupt" );if (panic_on_oops)Index:
linux/arch/x86/kernel/entry_32.S
===================================================================
arch/x86/kernel/entry_32.S line 84
Quoted 5 lines
arch/x86/kernel/entry_32.S +2
-0@@ -80,6 +80,8 @@ #define nr_syscalls ((syscall_table_size)/4) +#define HARDNMI_MASK 0x40000000 +
This is called "NMI_MASK" in 2.6.30. Did you test the x86_64 or x86_32 portion of this patch ? 64-bits seems ok, but not 32.
Quoted 7 lines
arch/x86/kernel/entry_32.S +0
-0#ifdef CONFIG_PREEMPT #define preempt_stop(clobbers) DISABLE_INTERRUPTS(clobbers); TRACE_IRQS_OFF #else@@ -344,8 +346,32 @@ END(ret_from_fork) # userspace resumption stub bypassing syscall exit tracing ALIGN RING0_PTREGS_FRAME
arch/x86/kernel/entry_32.S END(ret_from_fork) · line 358
Quoted 10 lines
arch/x86/kernel/entry_32.S +8
-0+ ret_from_exception: preempt_stop(CLBR_ANY) + GET_THREAD_INFO(%ebp) + movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS + movb PT_CS(%esp), %al + andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax + cmpl $USER_RPL, %eax + jae resume_userspace # returning to v8086 or userspace + testl $HARDNMI_MASK,TI_preempt_count(%ebp)
"NMI_MASK"
arch/x86/kernel/entry_32.S END(ret_from_fork) · line 360
Quoted 2 lines
arch/x86/kernel/entry_32.S +2
-0+ jz resume_kernel /* Not nested over NMI ? */ + testw $X86_EFLAGS_TF, PT_EFLAGS(%esp)
Hrm, I'm wondering if this test is not problematic for page faults. Basically, this test is there to deal with "single-stepping" in the nmi handler (it would not be "safe", but at least would not hang the system). So if page faults are raising the X86_EFLAGS_TF EFLAG, we definitely want to change this code. I am not sure about that, but it would be worth checking.
It's all I can spot for now, but if you have popf/ret firing to return to userspace instructions, there is clearly something fishy there.
Mathieu
Quoted 199 lines
arch/x86/kernel/entry_32.S +18
-0+ jnz resume_kernel /* + * If single-stepping an NMI handler, + * use the normal iret path instead of + * the popf/lret because lret would be + * single-stepped. It should not + * happen : it will reactivate NMIs + * prematurely. + */ + TRACE_IRQS_IRET + RESTORE_REGS + addl $4, %esp # skip orig_eax/error_code + CFI_ADJUST_CFA_OFFSET -4 + INTERRUPT_RETURN_NMI_SAFE + ret_from_intr: GET_THREAD_INFO(%ebp) check_userspace:@@ -851,6 +877,10 @@ ENTRY(native_iret) .previous END(native_iret) +ENTRY(native_nmi_return) + NATIVE_INTERRUPT_RETURN_NMI_SAFE # Should we deal with popf exception ? +END(native_nmi_return) + ENTRY(native_irq_enable_sysexit) sti sysexitIndex:
linux/arch/x86/kernel/entry_64.S
===================================================================arch/x86/kernel/entry_64.S +45
-12@@ -53,6 +53,7 @@ #include <asm/paravirt.h> #include <asm/ftrace.h> #include <asm/percpu.h> +#include <linux/hardirq.h> /* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */ #include <linux/elf-em.h>@@ -875,6 +876,9 @@ ENTRY(native_iret) .section __ex_table,"a" .quad native_iret, bad_iret .previous + +ENTRY(native_nmi_return) + NATIVE_INTERRUPT_RETURN_NMI_SAFE #endif .section .fixup,"ax"@@ -929,6 +933,23 @@ retint_signal: GET_THREAD_INFO(%rcx) jmp retint_with_reschedule + /* Returning to kernel space from exception. */ + /* rcx: threadinfo. interrupts off. */ +ENTRY(retexc_kernel) + testl $NMI_MASK, TI_preempt_count(%rcx) + jz retint_kernel /* Not nested over NMI ? */ + testw $X86_EFLAGS_TF, EFLAGS-ARGOFFSET(%rsp) /* trap flag? */ + jnz retint_kernel /* + * If single-stepping an NMI handler, + * use the normal iret path instead of + * the popf/lret because lret would be + * single-stepped. It should not + * happen : it will reactivate NMIs + * prematurely. + */ + RESTORE_ARGS 0, 8, 0 + INTERRUPT_RETURN_NMI_SAFE + #ifdef CONFIG_PREEMPT /* Returning to kernel space. Check if we need preemption */ /* rcx: threadinfo. interrupts off. */@@ -1407,34 +1428,46 @@ ENTRY(paranoid_exit) INTR_FRAME DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF - testl %ebx,%ebx /* swapgs needed? */ + testl %ebx, %ebx /* swapgs needed? */ jnz paranoid_restore - testl $3,CS(%rsp) + + testl $3, CS(%rsp) jnz paranoid_userspace + paranoid_swapgs: TRACE_IRQS_IRETQ 0 SWAPGS_UNSAFE_STACK RESTORE_ALL 8 jmp irq_return -paranoid_restore: +paranoid_restore_no_nmi: TRACE_IRQS_IRETQ 0 RESTORE_ALL 8 jmp irq_return +paranoid_restore: + GET_THREAD_INFO(%rcx) + testl $NMI_MASK, TI_preempt_count(%rcx) + jz paranoid_restore_no_nmi /* Nested over NMI ? */ + + testw $X86_EFLAGS_TF, EFLAGS-0(%rsp) /* trap flag? */ + jnz paranoid_restore_no_nmi + RESTORE_ALL 8 + INTERRUPT_RETURN_NMI_SAFE + paranoid_userspace: GET_THREAD_INFO(%rcx) - movl TI_flags(%rcx),%ebx - andl $_TIF_WORK_MASK,%ebx + movl TI_flags(%rcx), %ebx + andl $_TIF_WORK_MASK, %ebx jz paranoid_swapgs - movq %rsp,%rdi /* &pt_regs */ + movq %rsp, %rdi /* &pt_regs */ call sync_regs - movq %rax,%rsp /* switch stack for scheduling */ - testl $_TIF_NEED_RESCHED,%ebx + movq %rax, %rsp /* switch stack for scheduling */ + testl $_TIF_NEED_RESCHED, %ebx jnz paranoid_schedule - movl %ebx,%edx /* arg3: thread flags */ + movl %ebx, %edx /* arg3: thread flags */ TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) - xorl %esi,%esi /* arg2: oldset */ - movq %rsp,%rdi /* arg1: &pt_regs */ + xorl %esi, %esi /* arg2: oldset */ + movq %rsp, %rdi /* arg1: &pt_regs */ call do_notify_resume DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF@@ -1513,7 +1546,7 @@ ENTRY(error_exit) TRACE_IRQS_OFF GET_THREAD_INFO(%rcx) testl %eax,%eax - jne retint_kernel + jne retexc_kernel LOCKDEP_SYS_EXIT_IRQ movl TI_flags(%rcx),%edx movl $_TIF_WORK_MASK,%ediIndex:
linux/include/linux/hardirq.h
===================================================================include/linux/hardirq.h +11
-5@@ -1,12 +1,14 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#ifndef __ASSEMBLY__#include <linux/preempt.h> #include <linux/smp_lock.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h> #include <asm/hardirq.h> #include <asm/system.h> +#endif /* * We put the hardirq and softirq counter into the preemption @@ -50,17 +52,17 @@#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) -#define __IRQ_MASK (x) ((1UL << (x))-1) +#define __IRQ_MASK (x) ((1 << (x))-1)#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) -#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT) -#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) -#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) -#define NMI_OFFSET (1UL << NMI_SHIFT) +#define PREEMPT_OFFSET (1 << PREEMPT_SHIFT) +#define SOFTIRQ_OFFSET (1 << SOFTIRQ_SHIFT) +#define HARDIRQ_OFFSET (1 << HARDIRQ_SHIFT) +#define NMI_OFFSET (1 << NMI_SHIFT)#if PREEMPT_ACTIVE < (1 << (NMI_SHIFT +NMI_BITS ))#error PREEMPT_ACTIVE is too low!@@ -116,6 +118,8 @@# define IRQ_EXIT_OFFSET HARDIRQ_OFFSET#endif +#ifndef __ASSEMBLY__ +#if defined(CONFIG_SMP ) || defined(CONFIG_GENERIC_HARDIRQS )extern void synchronize_irq (unsigned int irq);#else @@ -195,4 +199,6 @@ extern void irq_exit(void);ftrace_nmi_exit (); \ } while (0) +#endif /* !__ASSEMBLY__ */ +#endif /* LINUX_HARDIRQ_H */
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 244 lines · Mathieu Desnoyers
Quoted 243 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 32 lines · Mathieu Desnoyers
Quoted 24 lines · * tip-bot for Peter Zijlstra ([email protected]) wrote:
Commit-ID: 74193ef0ecab92535c8517f082f1f50504526c9b Gitweb: http://git.kernel.org/tip/74193ef0ecab92535c8517f082f1f50504526c9b Author: Peter Zijlstra <[email protected]> AuthorDate: Mon, 15 Jun 2009 13:07:24 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Jun 2009 15:57:53 +0200
perf_counter: x86: Fix call-chain support to use NMI-safe methods
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.Therefore use a GUP based approach to copy the stack frames over.
We tried an alternative solution as well: we used a forward ported version of Mathieu Desnoyers's "NMI safe INT3 and Page Fault" patch that modifies the exception return path to use an open-coded IRET with explicit stack unrolling and TF checking.
This didnt work as it interacted with faulting user-space instructions, causing them not to restart properly, which corrupts user-space registers.
Quoted 6 lines
Solving that would probably involve disassembling those instructions and backtracing the RIP. But even without that, the code was deemed rather complex to the already non-trivial x86 entry assembly code, so instead we went for this GUP based method that does a software-walk of the pagetables.
Quoted 5 lines
Hrm, I'm probably missing something. Normally, you should test for "
in_nmi()" upon return from exception, and only in these cases go for the open-coded IRET with stack unrolling and ret. I really don't see how you end up messing up the page fault return to userspace path, as it's impossible to havein_nmi()set.here's the (heavily modified) version of your patch that i used.
Ingo
-------------------->
Subject: x86 NMI-safe INT3 and Page Fault
From: Mathieu Desnoyers <[email protected]>
Date: Mon, 12 May 2008 21:21:07 +0200Implements an alternative iret with popf and return so trap and exception handlers can return to the NMI handler without issuing iret. iret would cause NMIs to be reenabled prematurely.
x86_32uses popf and far return.x86_64has to copy the return instruction pointer to the top of the previous stack, issue a popf, loads the previous esp and issue a near return (ret).It allows placing immediate values (and therefore optimized
trace_marks) in NMI code since returning from a breakpoint would be valid. Accessing vmalloc'd memory, which allows executing module code or accessing vmapped or vmalloc'd areas from NMI context, would also be valid. This is very useful to tracers like LTTng.This patch makes all faults, traps and exception safe to be called from NMI context *except* single-stepping, which requires iret to restore the TF (trap flag) and jump to the return address in a single instruction. Sorry, no kprobes support in NMI handlers because of this limitation. We cannot single-step an NMI handler, because iret must set the TF flag and return back to the instruction to single-step in a single instruction. This cannot be emulated with popf/lret, because lret would be single-stepped. It does not apply to immediate values because they do not use single-stepping. This code detects if the TF flag is set and uses the iret path for single-stepping, even if it reactivates NMIs prematurely.
Test to detect if nested under a NMI handler is only done upon the return from trap/exception to kernel, which is not frequent. Other return paths (return from trap/exception to userspace, return from interrupt) keep the exact same behavior (no slowdown).
Depends on :
change-alpha-active-count-bit.patch
change-avr32-active-count-bit.patchTODO : test with lguest, xen, kvm.
** This patch depends on the "Stringify support commas" patchset ** ** Also depends on fix-x86_64-page-fault-scheduler-race patch **
tested on
x86_32(tests implemented in a separate patch) :
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
- tested vmalloc faults in NMI handler by placing a non-optimized marker in the NMI handler (so no breakpoint is executed) and connecting a probe which touches every pages of a 20MB vmalloc'd buffer. It executes trough the return path without problem.
- Tested with and without preemptiontested on
x86_64
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.To test on
x86_64:
- Test without preemption
- Test vmalloc faults
- Test on Intel 64 bits CPUs. (AMD64 was fine)Changelog since v1 :
-x86_64fixes.
Changelog since v2 :
- fix paravirt build
Changelog since v3 :
- Include modifications suggested by Jeremy
Changelog since v4 :
- includinghardirq.hin entry_32/64.S is a bad idea (non ifndef'd C code), defineHARDNMI_MASKin the .S files directly.
Changelog since v5 :
- AddHARDNMI_MASKtoirq_count()and makedie()more verbose for NMIs. Changelog since v7 :
- Implement paravirtualizednmi_return.
Changelog since v8 :
- refreshed the patch for asm-offsets. Those were left out of v8.
- now depends on "Stringify support commas" patch.
Changelog since v9 :
- Only test the nmi nested preempt count flag upon return from exceptions, not on return from interrupts. Only the kernel return path has this test.
- Add Xen, VMI, lguest support. Use their iret pavavirt ops in lieu ofnmi_return.-- Ported to sched-devel.git
Signed-off-by Mathieu Desnoyers <[email protected]>CC "H. Peter Anvin" <[email protected]>CC Jeremy Fitzhardinge <[email protected]>CC Steven Rostedt <[email protected]>CC "Frank Ch. Eigler" <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/include/asm/irqflags.h | 56 +++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/dumpstack.c | 2 + arch/x86/kernel/entry_32.S | 30 +++++++++++++++++++++ arch/x86/kernel/entry_64.S | 57 +++++++++++++++++++++++++++++++--------- include/linux/hardirq.h | 16 +++++++---- 5 files changed, 144 insertions(+), 17 deletions(-)Index:
linux/arch/x86/include/asm/irqflags.h
===================================================================arch/x86/include/asm/irqflags.h +56
-0@@ -51,6 +51,61 @@ static inline void native_halt(void)#endif +#ifdef CONFIG_X86_64 +/* +* Only returns from a trap or exception to a NMI context (intra-privilege +* level near return) to the same SS and CS segments. Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. It takes care of modifying the eflags, rsp and returning to the +* previous function. +* +* The stack, at that point, looks like : +* +* 0(rsp) RIP +* 8(rsp) CS +* 16(rsp) EFLAGS +* 24(rsp) RSP +* 32(rsp) SS +* +* Upon execution : +* Copy EIP to the top of the return stack +* Update top of return stack address +* Pop eflags into the eflags register +* Make the return stack current +* Near return (popping the return address from the return stack) +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret +#else +/* +* Protected mode only, no V8086. Implies that protected mode must +* be entered before NMIs or MCEs are enabled. Only returns from a trap or +* exception to a NMI context (intra-privilege level far return). Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. +* +* The stack, at that point, looks like : +* +* 0(esp) EIP +* 4(esp) CS +* 8(esp) EFLAGS +* +* Upon execution : +* Copy the stack eflags to top of stack +* Pop eflags into the eflags register +* Far return: pop EIP and CS into their register, and additionally pop EFLAGS. +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushl 8(%esp); \ + popfl; \ + lret $4 +#endif +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else @@ -109,6 +164,7 @@ static inline unsigned long __raw_local_#define ENABLE_INTERRUPTS (x) sti#define DISABLE_INTERRUPTS (x) cli +#define INTERRUPT_RETURN_NMI_SAFE NATIVE_INTERRUPT_RETURN_NMI_SAFE#ifdef CONFIG_X86_64 #define SWAPGS swapgsIndex:
linux/arch/x86/kernel/dumpstack.c
===================================================================arch/x86/kernel/dumpstack.c +2
-0@@ -237,6 +237,8 @@ void __kprobes oops_end(unsigned long flif (!signr)return ; +if (in_nmi ()) +panic ("Fatal exception in non-maskable interrupt" );if (in_interrupt ())panic ("Fatal exception in interrupt" );if (panic_on_oops)Index:
linux/arch/x86/kernel/entry_32.S
===================================================================arch/x86/kernel/entry_32.S +0
-0@@ -80,6 +80,8 @@
Quoted 7 lines
arch/x86/kernel/entry_32.S line 84
Quoted 4 lines
arch/x86/kernel/entry_32.S +2
-0#define nr_syscalls ((syscall_table_size)/4) +#define HARDNMI_MASK 0x40000000 +This is called "
NMI_MASK" in 2.6.30. Did you test thex86_64orx86_32portion of this patch ? 64-bits seems ok, but not 32.
i only tested the 64-bit side, and fixed up NMI_MASK only on that side (as you can see it from the patch). This was a partial port of your patch.
Quoted 3 lines
It's all I can spot for now, but if you have popf/ret firing to return to userspace instructions, there is clearly something fishy there.
I think Linus's observation about cr2 corruption explains all the symptoms i saw.
And it all stabilized and started behaving under load once we switched to Peter's fast-GUP based soft-pte-lookup method.
Ingo
Quoted 4 lines · tip-bot for Peter Zijlstra
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.
That's not the only problem.
An even more fundamental problem is that the page fault handler is not re-entrant because of simple the value in %cr2. So regardless of any 'iret' issues, you *CANNOT* take a page fault in an NMI, because the NMI might happen while we're in the critical region of having taken another page fault, but before we've saved off the value of %cr2 in that old page fault.
If the NMI handler causes a page fault, it will corrupt the %cr2 of the outer page fault. That's why the page fault is done with an interrupt gate, and why we have that conditional local_irq_enable() in it.
So page faults are fundamentally only safe wrt normal interrupts, not NMI.
Linus
Quoted 15 lines · Linus Torvalds
Quoted 4 lines · tip-bot for Peter Zijlstra
__copy_from_user_inatomic()isn't NMI safe in that it can trigger the page fault handler which is another trap and its return path invokes IRET which will also close the NMI context.That's not the only problem.
An even more fundamental problem is that the page fault handler is not re-entrant because of simple the value in %cr2. So regardless of any 'iret' issues, you *CANNOT* take a page fault in an NMI, because the NMI might happen while we're in the critical region of having taken another page fault, but before we've saved off the value of %cr2 in that old page fault.
Quoted 7 lines
If the NMI handler causes a page fault, it will corrupt the %cr2 of the outer page fault. That's why the page fault is done with an interrupt gate, and why we have that conditional
local_irq_enable()in it.So page faults are fundamentally only safe wrt normal interrupts, not NMI.
ahhh ... a light goes up. Indeed.
I was suspecting something much more complex: like the CPU somehow having shadow state for attempted-fault which gets confused by NMI->fault.
A simple cr2 corruption would explain all those cc1 SIGSEGVs and other user-space crashes i saw, with sufficiently intense sampling - easily.
The thing is, that __copy_user_inatomic() has been in arch/x86/oprofile/backtrace.c for years, i didnt even suspect some simple, fundamental flaw like this. Apparently nobody uses it.
This is really good news in a sense: i really hate that additional entry*.S mucking in the exception path in the dont-IRET patch. We want less entry*.S magic, not more.
Ingo
Quoted 4 lines · Ingo Molnar
A simple cr2 corruption would explain all those cc1 SIGSEGVs and other user-space crashes i saw, with sufficiently intense sampling - easily.
Note that we could work around the %cr2 issue, since any corruption is always nicely "nested" (ie there are never any SMP issues with async writes to the register).
So what we _could_ do is to have a magic value for %cr2, along with a "NMI sequence count", and if we see that value, we just return (without doing anything) from the page fault handler.
Then, the NMI handler would be changed to always write that value to %cr2 after it has done the operation that could fault, and do an atomic increment of the NMI sequence count. Then, we can do something like this in the page fault handler:
if (cr2 == MAGIC_CR2) {
static unsigned long my_seqno = -1;
if (my_seqno != nmi_seqno) {
my_seqno = nmi_seqno;
return;
}
}where the whole (and only) point of that "seqno" is to protect against user space doing something like
int i = *(int *)MAGIC_CR2;
and causing infinite faults.
If a real NMI happens, then nmi_seqno will always be different, and we'll just retry the fault (the NMI handler would do something like
write_cr2(MAGIC_CR2); atomic_inc(&nmi_seqno);
to set it all up).
Anyway, I do think that the _correct_ solution is to not do page faults from within NMI's, but the above is an outline of how we could _try_ to handle it if we really really wanted to. IOW, the fact that cr2 gets corrupted is not insurmountable, exactly because we _could_ always just retrigger the page fault, and thus "re-create' the corrupted %cr2 value.
Hacky, hacky. And I'm not sure how happy CPU's even are to have %cr2 written to, so we could hit CPU issues.
Linus
Quoted 13 lines · Linus Torvalds
Then, the NMI handler would be changed to always write that value to %cr2 after it has done the operation that could fault, and do an atomic increment of the NMI sequence count. Then, we can do something like this in the page fault handler:
if (cr2 == MAGIC_CR2) { static unsigned long my_seqno = -1; if (my_seqno != nmi_seqno) { my_seqno = nmi_seqno; return; } }
Quoted 6 lines
where the whole (and only) point of that "seqno" is to protect against user space doing something like
int i = *(int *)MAGIC_CR2;and causing infinite faults.
Heh - this is so tricky that it's disgusting! Lovely.
And, since this appears to be a competition of sick ideas, an even more disgusting hack might be to write to the IDT from the NMI handler, and install a NULL entry at #PF and rely on the double fault handler to detect faults - double faults dont clobber the cr2 i think ...
( I think to protect the fragile and pure fabric of lkml against
moral corruption, disgusting patches must remain unsent and disgusting ideas like this must absolutely stay unspoken. Hence i have removed lkml from the Cc:. [Oops i didnt ... too late, and this mail has already been sent! :-/ ])
Quoted 16 lines
If a real NMI happens, then
nmi_seqnowill always be different, and we'll just retry the fault (the NMI handler would do something likewrite_cr2(MAGIC_CR2); atomic_inc(&nmi_seqno);to set it all up).
Anyway, I do think that the _correct_ solution is to not do page faults from within NMI's, but the above is an outline of how we could _try_ to handle it if we really really wanted to. IOW, the fact that cr2 gets corrupted is not insurmountable, exactly because we _could_ always just retrigger the page fault, and thus "re-create' the corrupted %cr2 value.
Quoted 2 lines
Hacky, hacky. And I'm not sure how happy CPU's even are to have %cr2 written to, so we could hit CPU issues.
If cr2 cannot be safely written to on a CPU, that could be worked around by counting the number of NMIs via a percpu_add(this_nmi_count, 1) and retrying faults if any NMI happened between the previous fault and this fault.
This has the disadvantage of potentially doubling the number of pagefaults though. But it would certainly work as a tricky quirk to this quirk which is added to a rather quirky code-path to begin with.
Ingo
Quoted 44 lines · * Linus Torvalds ([email protected]) wrote:
Quoted 4 lines · Ingo Molnar
A simple cr2 corruption would explain all those cc1 SIGSEGVs and other user-space crashes i saw, with sufficiently intense sampling - easily.
Note that we could work around the %cr2 issue, since any corruption is always nicely "nested" (ie there are never any SMP issues with async writes to the register).
So what we _could_ do is to have a magic value for %cr2, along with a "NMI sequence count", and if we see that value, we just return (without doing anything) from the page fault handler.
Then, the NMI handler would be changed to always write that value to %cr2 after it has done the operation that could fault, and do an atomic increment of the NMI sequence count. Then, we can do something like this in the page fault handler:
if (cr2 == MAGIC_CR2) { static unsigned long my_seqno = -1; if (my_seqno != nmi_seqno) { my_seqno = nmi_seqno; return; } }where the whole (and only) point of that "seqno" is to protect against user space doing something like
int i = *(int *)MAGIC_CR2;and causing infinite faults.
If a real NMI happens, then
nmi_seqnowill always be different, and we'll just retry the fault (the NMI handler would do something likewrite_cr2(MAGIC_CR2); atomic_inc(&nmi_seqno);to set it all up).
Quoted 9 lines
Anyway, I do think that the _correct_ solution is to not do page faults from within NMI's, but the above is an outline of how we could _try_ to handle it if we really really wanted to. IOW, the fact that cr2 gets corrupted is not insurmountable, exactly because we _could_ always just retrigger the page fault, and thus "re-create' the corrupted %cr2 value.
Hacky, hacky. And I'm not sure how happy CPU's even are to have %cr2 written to, so we could hit CPU issues.
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ? This would ensure a nmi-interrupted page fault handler would continue what it was doing with a non-corrupted cr2 register after returning from nmi.
Plus, this involves no modification to the page fault handler fast path.
But I fear I might be missing something totally obvious.
Mathieu
Quoted 1 line
Linus
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 7 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ? This would ensure a nmi-interrupted page fault handler would continue what it was doing with a non-corrupted cr2 register after returning from nmi.
Plus, this involves no modification to the page fault handler fast path.
I guess this kind of nesting would work too - assuming the cr2 can be written to robustly.
And i suspect CPU makers pull off a few tricks to stage the cr2 info away from the page fault entry execution asynchronously, so i'd not be surprised if writing to it uncovered unknown-so-far side-effects in CPU implementations.
If possible i wouldnt want to rely on such a narrowly possible hack really - any small change in CPU specs could cause problems years down the line.
The GUP based method is pretty generic though - and can be used on other architectures as well. It's not as fast as direct access though.
Ingo
Quoted 3 lines · Ingo Molnar
The GUP based method is pretty generic though - and can be used on other architectures as well. It's not as fast as direct access though.
Another question is: your patch switches over all normal exceptions from IRET to hand-unroll+RET.
It would be really nice to benchmark it (via 'perf stat' for example ;-) whether that's a slowdown or a speedup.
If it's a slowdown then the decision is easy: we dont want this, we want to push the overhead into the sampling code, away from common codepaths.
[ If on the other hand it's a speedup of a few cycles then we have
the problem of me suddenly liking this patch a whole lot more ;-) ]
Ingo
Quoted 10 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 3 lines · Ingo Molnar
The GUP based method is pretty generic though - and can be used on other architectures as well. It's not as fast as direct access though.
Another question is: your patch switches over all normal exceptions from IRET to hand-unroll+RET.
Nope, it actually only switches the exceptions returning from an exception handler nested in NMI context to the hand-unroll+RET version. Given such exception nesting is expected to be very rare, it should not show any performance difference.
I also organised the code to make sure I did not add any test to the fast paths in my original patch.
Quoted 7 lines
It would be really nice to benchmark it (via 'perf stat' for example ;-) whether that's a slowdown or a speedup.
If it's a slowdown then the decision is easy: we dont want this, we want to push the overhead into the sampling code, away from common codepaths.
I did not try to make the "hand unroll + ret" the default. I therefore don't know if it is faster or slower than iret. But I prefered to stay on the safe side and only modify the exceptions nested within NMI handlers.
Mathieu
Quoted 4 lines
[ If on the other hand it's a speedup of a few cycles then we have
the problem of me suddenly liking this patch a whole lot more ;-) ]Ingo
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 8 lines · Mathieu Desnoyers
Quoted 3 lines
If it's a slowdown then the decision is easy: we dont want this, we want to push the overhead into the sampling code, away from common codepaths.
I did not try to make the "hand unroll + ret" the default. I therefore don't know if it is faster or slower than iret. But I prefered to stay on the safe side and only modify the exceptions nested within NMI handlers.
hm, i misread that bit then. Too bad.
Btw., that speedup question is still valid. (Just not relevant here and now.)
Ingo
Quoted 3 lines · Ingo Molnar
[ If on the other hand it's a speedup of a few cycles then we have
the problem of me suddenly liking this patch a whole lot more ;-) ]
I missed the patch.
It's quite possible that replacing "iret" with a regular "ret" (for the kernel->kernel transition) is a real speedup. That said, there's a few things to think about:
- CPU return stack caches/predictors. I suspect that "iret" and exceptions don't generally touch them (but who knows - maybe they do), while a regular "ret" definitely does. I dunno about "retf".
This can cause very subtle performance slowdowns, where the slowdown happens somewhere else. And it could be _very_ uarch-dependent (ie only happen on some architectures, while having no performance downside on others)
- kernel->kernel exceptions _should_ be rare, with the exception of actual real external interrupts. So the path to optimize should always be the user-space exception path. That one will need 'iret', but I'd also not want to see more testing in that hot-path. I suspect we already always test for user-mode anyway (due to signal handling etc work), but if it adds new tests to that path, any kernel->kernel speedup is likely totally pointless.
That said, it would be nice to avoid 'iret' if only because of its subtle interactions with the while NMI flag.
Linus
Quoted 9 lines · * Linus Torvalds ([email protected]) wrote:
Quoted 3 lines · Ingo Molnar
[ If on the other hand it's a speedup of a few cycles then we have
the problem of me suddenly liking this patch a whole lot more ;-) ]I missed the patch.
Quoted 8 lines
It's quite possible that replacing "iret" with a regular "ret" (for the
kernel->kerneltransition) is a real speedup. That said, there's a few things to think about:- CPU return stack caches/predictors. I suspect that "iret" and exceptions don't generally touch them (but who knows - maybe they do), while a regular "ret" definitely does. I dunno about "retf".
Also, I don't think popf acts as a serializing instruction. (reading the Intel Architecture Software Developer's Manual, vol 3 System programming section 7.4 Serializing Instructions). And ret clearly doesn't. Therefore I'd be worried of iret serialization assumptions which could be done within the kernel code. Maybe it would be safer to add a cpuid or mfence instruction in there to ensure correct core serialization.
This would turn the hand-made iret into a :
popf
mfence
ret
Or something like that.
Mathieu
Quoted 17 lines
This can cause very subtle performance slowdowns, where the slowdown happens somewhere else. And it could be _very_ uarch-dependent (ie only happen on some architectures, while having no performance downside on others)
-
kernel->kernelexceptions _should_ be rare, with the exception of actual real external interrupts. So the path to optimize should always be the user-space exception path. That one will need 'iret', but I'd also not want to see more testing in that hot-path. I suspect we already always test for user-mode anyway (due to signal handling etc work), but if it adds new tests to that path, anykernel->kernelspeedup is likely totally pointless.That said, it would be nice to avoid 'iret' if only because of its subtle interactions with the while NMI flag.
Linus
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 23 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 7 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ? This would ensure a nmi-interrupted page fault handler would continue what it was doing with a non-corrupted cr2 register after returning from nmi.
Plus, this involves no modification to the page fault handler fast path.
I guess this kind of nesting would work too - assuming the cr2 can be written to robustly.
And i suspect CPU makers pull off a few tricks to stage the cr2 info away from the page fault entry execution asynchronously, so i'd not be surprised if writing to it uncovered unknown-so-far side-effects in CPU implementations.
If possible i wouldnt want to rely on such a narrowly possible hack really - any small change in CPU specs could cause problems years down the line.
Quoted 5 lines
The GUP based method is pretty generic though - and can be used on other architectures as well. It's not as fast as direct access though.
Ingo
I guess. However, having the ability to call module code in NMI handler context without having to fear for page fault handler re-entrancy (on x86 32) seems like an interesting overall simplification of nmi-handler rules. It is currently far from trivial to write code aimed at NMI handler context. I mean.. LTTng should not have to run vmalloc_sync_all() after loading its modules as it currently does.
Maybe it would be worth trying the save/restore cr2 approach and test to figure out how a large variety of machines react. The fact is that hypervisor code already writes into the cr2 register :
kvm/vmx.c :vmx_vcpu_run()
...
"mov %%"R"ax, %%cr2 \n\t"
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 2 lines · Mathieu Desnoyers
I guess. However, having the ability to call module code in NMI handler context [...]
I dont think we really want to execute module-provided code from NMI handlers. NMI context is pretty tricky to begin with. (Yes, i know it's possible via nmi notifiers or modular oprofile but it's a really bad practice IMHO.)
Ingo
H. Peter Anvin
reviewerQuoted 10 lines · Ingo Molnar
Quoted 7 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ? This would ensure a nmi-interrupted page fault handler would continue what it was doing with a non-corrupted cr2 register after returning from nmi.
Plus, this involves no modification to the page fault handler fast path.
Quoted 8 lines
I guess this kind of nesting would work too - assuming the cr2 can be written to robustly.
And i suspect CPU makers pull off a few tricks to stage the cr2 info away from the page fault entry execution asynchronously, so i'd not be surprised if writing to it uncovered unknown-so-far side-effects in CPU implementations.
I wouldn't actually expect that, *as long as* there is serialization between the cr2 write and the cr2 read.
-hpa
Quoted 14 lines · H. Peter Anvin
Quoted 13 lines · Ingo Molnar
Quoted 7 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ? This would ensure a nmi-interrupted page fault handler would continue what it was doing with a non-corrupted cr2 register after returning from nmi.
Plus, this involves no modification to the page fault handler fast path.
I guess this kind of nesting would work too - assuming the cr2 can be written to robustly.
Quoted 8 lines
Quoted 5 lines
And i suspect CPU makers pull off a few tricks to stage the cr2 info away from the page fault entry execution asynchronously, so i'd not be surprised if writing to it uncovered unknown-so-far side-effects in CPU implementations.
I wouldn't actually expect that, *as long as* there is serialization between the cr2 write and the cr2 read.
Well, is there any OS that heavily relies on cr2 writes and which uses them from NMI context, and which CPU makers care about? (Meaning: Windows, pretty much.)
If not then i agree that in theory it should work fine, but in practice we only know that we dont know the unknown risk here ;-)
Ingo
H. Peter Anvin
reviewerQuoted 11 lines · Ingo Molnar
Quoted 3 lines
Quoted 1 line
I wouldn't actually expect that, *as long as* there is serialization between the cr2 write and the cr2 read.
Well, is there any OS that heavily relies on cr2 writes and which uses them from NMI context, and which CPU makers care about? (Meaning: Windows, pretty much.)
If not then i agree that in theory it should work fine, but in practice we only know that we dont know the unknown risk here ;-)
I think you can drop "uses them from NMI context" from that statement; writing to %cr2 is independent of the context.
I can try to find out internally what Intel's position on writing %cr2 is, but it'll take a while; however, KVM should be able to tell you if any random OS uses %cr2 writes (as should a static disassembly of their kernel.)
-hpa
Quoted 14 lines · H. Peter Anvin
Quoted 11 lines · Ingo Molnar
Quoted 3 lines
I wouldn't actually expect that, *as long as* there is serialization between the cr2 write and the cr2 read.
Well, is there any OS that heavily relies on cr2 writes and which uses them from NMI context, and which CPU makers care about? (Meaning: Windows, pretty much.)
If not then i agree that in theory it should work fine, but in practice we only know that we dont know the unknown risk here ;-)
Quoted 8 lines
I think you can drop "uses them from NMI context" from that statement; writing to %cr2 is independent of the context.
I can try to find out internally what Intel's position on writing %cr2 is, but it'll take a while; however, KVM should be able to tell you if any random OS uses %cr2 writes (as should a static disassembly of their kernel.)
Linux is one such OS. When acting as a hypervisor it writes cr2 to present its guests with their expected environment (any hypervisor that uses virtualization extensions will of course need to do this).
Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Quoted 23 lines · Avi Kivity
Quoted 14 lines · H. Peter Anvin
Quoted 11 lines · Ingo Molnar
Quoted 3 lines
I wouldn't actually expect that, *as long as* there is serialization between the cr2 write and the cr2 read.
Well, is there any OS that heavily relies on cr2 writes and which uses them from NMI context, and which CPU makers care about? (Meaning: Windows, pretty much.)
If not then i agree that in theory it should work fine, but in practice we only know that we dont know the unknown risk here ;-)
Quoted 7 lines
I think you can drop "uses them from NMI context" from that statement; writing to %cr2 is independent of the context.
I can try to find out internally what Intel's position on writing %cr2 is, but it'll take a while; however, KVM should be able to tell you if any random OS uses %cr2 writes (as should a static disassembly of their kernel.)
Quoted 4 lines
Linux is one such OS. When acting as a hypervisor it writes cr2 to present its guests with their expected environment (any hypervisor that uses virtualization extensions will of course need to do this).
Ah, it does save/restore it in svm_vcpu_run. VMX can do this via its context structure (without explicit CR manipulations in host space), right?
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 11 lines
Quoted 6 lines
I can try to find out internally what Intel's position on writing %cr2 is, but it'll take a while; however, KVM should be able to tell you if any random OS uses %cr2 writes (as should a static disassembly of their kernel.)
Linux is one such OS. When acting as a hypervisor it writes cr2 to present its guests with their expected environment (any hypervisor that uses virtualization extensions will of course need to do this).
Quoted 4 lines
Ah, it does save/restore it in
svm_vcpu_run. VMX can do this via its context structure (without explicit CR manipulations in host space), right?
It's the other way around. svm switches the guest cr2 in hardware (through svm->vmcb->save.cr2). The code you're referring to saves and restores the host cr2, which is completely unnecessary. I'm currently in the middle of dropping it :)
vmx has no hardware support for switching cr2, so vmx_vcpu_run() switches it using mov cr2. Given that it's pretty expensive, I've switched it to write-if-changed, which dropped 70 cycles from the vmexit latency.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
Quoted 12 lines · Ingo Molnar
Quoted 11 lines
Quoted 6 lines
I can try to find out internally what Intel's position on writing %cr2 is, but it'll take a while; however, KVM should be able to tell you if any random OS uses %cr2 writes (as should a static disassembly of their kernel.)
Linux is one such OS. When acting as a hypervisor it writes cr2 to present its guests with their expected environment (any hypervisor that uses virtualization extensions will of course need to do this).
Quoted 4 lines
Ah, it does save/restore it in
svm_vcpu_run. VMX can do this via its context structure (without explicit CR manipulations in host space), right?
Quoted 4 lines
It's the other way around. svm switches the guest cr2 in hardware (through
svm->vmcb->save.cr2). The code you're referring to saves and restores the host cr2, which is completely unnecessary. I'm currently in the middle of dropping it :)
Heh :)
Quoted 4 lines
vmx has no hardware support for switching cr2, so
vmx_vcpu_run()switches it using mov cr2. Given that it's pretty expensive, I've switched it to write-if-changed, which dropped 70 cycles from the vmexit latency.
Yep, see my numbers elsewhere in this thread - the cost of a cr2 write is ~84 cycles on Nehalem.
Ingo
Quoted 3 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ?
Yes, that would work as well, and be less subtle.
It still does have the same worries about CPU's not being all that happy about writing to %cr2 (we do it when restoring CPU state at resume time, but nobody has ever _cared_ before, so I don't know if it matters).
Linus
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ?
Yes, that would work as well, and be less subtle.
It still does have the same worries about CPU's not being all that happy about writing to %cr2 (we do it when restoring CPU state at resume time, but nobody has ever _cared_ before, so I don't know if it matters).
I think we can dodge the whole issue by asking whether the old 1-year-old patch from Mathieu (repeated below - partially ported and barely tested / not signed off) is an actual speedup in the normal exception codepaths.
The gist of it is the replacement of iret with this open-coded sequence:
+#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \
+ movq %rsp, %rax; \
+ movq 24+8(%rax), %rsp; \
+ pushq 0+8(%rax); \
+ pushq 16+8(%rax); \
+ movq (%rax), %rax; \
+ popfq; \
+ ret
Whether popfq+ret is faster than iret is the question i think. (beyond the question of 'how about all the weird stack exceptions that are possible above')
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
If it's slower then it's a non-starter and we do the GUP solution.
Lets hope it's slower ;-)
Ingo
----------->
Subject: x86 NMI-safe INT3 and Page Fault
From: Mathieu Desnoyers <[email protected]>
Date: Mon, 12 May 2008 21:21:07 +0200
Implements an alternative iret with popf and return so trap and exception handlers can return to the NMI handler without issuing iret. iret would cause NMIs to be reenabled prematurely. x86_32 uses popf and far return. x86_64 has to copy the return instruction pointer to the top of the previous stack, issue a popf, loads the previous esp and issue a near return (ret).
It allows placing immediate values (and therefore optimized trace_marks) in NMI code since returning from a breakpoint would be valid. Accessing vmalloc'd memory, which allows executing module code or accessing vmapped or vmalloc'd areas from NMI context, would also be valid. This is very useful to tracers like LTTng.
This patch makes all faults, traps and exception safe to be called from NMI context *except* single-stepping, which requires iret to restore the TF (trap flag) and jump to the return address in a single instruction. Sorry, no kprobes support in NMI handlers because of this limitation. We cannot single-step an NMI handler, because iret must set the TF flag and return back to the instruction to single-step in a single instruction. This cannot be emulated with popf/lret, because lret would be single-stepped. It does not apply to immediate values because they do not use single-stepping. This code detects if the TF flag is set and uses the iret path for single-stepping, even if it reactivates NMIs prematurely.
Test to detect if nested under a NMI handler is only done upon the return from trap/exception to kernel, which is not frequent. Other return paths (return from trap/exception to userspace, return from interrupt) keep the exact same behavior (no slowdown).
Depends on :
change-alpha-active-count-bit.patch
change-avr32-active-count-bit.patch
TODO : test with lguest, xen, kvm.
** This patch depends on the "Stringify support commas" patchset ** ** Also depends on fix-x86_64-page-fault-scheduler-race patch **
tested on x86_32 (tests implemented in a separate patch) :
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
- trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
- tested vmalloc faults in NMI handler by placing a non-optimized marker in the NMI handler (so no breakpoint is executed) and connecting a probe which touches every pages of a 20MB vmalloc'd buffer. It executes trough the return path without problem.
- Tested with and without preemption
tested on x86_64
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
- trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
To test on x86_64 :
- Test without preemption
- Test vmalloc faults
- Test on Intel 64 bits CPUs. (AMD64 was fine)
Changelog since v1 :
- x86_64 fixes.
Changelog since v2 :
- fix paravirt build
Changelog since v3 :
- Include modifications suggested by Jeremy
Changelog since v4 :
- including hardirq.h in entry_32/64.S is a bad idea (non ifndef'd C code), define HARDNMI_MASK in the .S files directly.
Changelog since v5 :
- Add HARDNMI_MASK to irq_count() and make die() more verbose for NMIs. Changelog since v7 :
- Implement paravirtualized nmi_return.
Changelog since v8 :
- refreshed the patch for asm-offsets. Those were left out of v8.
- now depends on "Stringify support commas" patch.
Changelog since v9 :
- Only test the nmi nested preempt count flag upon return from exceptions, not on return from interrupts. Only the kernel return path has this test.
- Add Xen, VMI, lguest support. Use their iret pavavirt ops in lieu of nmi_return.
-- Ported to sched-devel.git
--- arch/x86/include/asm/irqflags.h | 56 +++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/dumpstack.c | 2 + arch/x86/kernel/entry_64.S | 57 +++++++++++++++++++++++++++++++--------- include/linux/hardirq.h | 16 +++++++----
arch/x86/include/asm/irqflags.h +56 -0
@@ -51,6 +51,61 @@ static inline void native_halt(void)#endif +#ifdef CONFIG_X86_64 +/* +* Only returns from a trap or exception to a NMI context (intra-privilege +* level near return) to the same SS and CS segments. Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. It takes care of modifying the eflags, rsp and returning to the +* previous function. +* +* The stack, at that point, looks like : +* +* 0(rsp) RIP +* 8(rsp) CS +* 16(rsp) EFLAGS +* 24(rsp) RSP +* 32(rsp) SS +* +* Upon execution : +* Copy EIP to the top of the return stack +* Update top of return stack address +* Pop eflags into the eflags register +* Make the return stack current +* Near return (popping the return address from the return stack) +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret +#else +/* +* Protected mode only, no V8086. Implies that protected mode must +* be entered before NMIs or MCEs are enabled. Only returns from a trap or +* exception to a NMI context (intra-privilege level far return). Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. +* +* The stack, at that point, looks like : +* +* 0(esp) EIP +* 4(esp) CS +* 8(esp) EFLAGS +* +* Upon execution : +* Copy the stack eflags to top of stack +* Pop eflags into the eflags register +* Far return: pop EIP and CS into their register, and additionally pop EFLAGS. +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushl 8(%esp); \ + popfl; \ + lret $4 +#endif +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else
@@ -109,6 +164,7 @@ static inline unsigned long __raw_local_#define ENABLE_INTERRUPTS (x) sti#define DISABLE_INTERRUPTS (x) cli +#define INTERRUPT_RETURN_NMI_SAFE NATIVE_INTERRUPT_RETURN_NMI_SAFE#ifdef CONFIG_X86_64 #define SWAPGS swapgs
arch/x86/kernel/dumpstack.c +2 -0
@@ -237,6 +237,8 @@ void __kprobes oops_end(unsigned long flif (!signr)return ; +if (in_nmi ()) +panic ("Fatal exception in non-maskable interrupt" );if (in_interrupt ())panic ("Fatal exception in interrupt" );if (panic_on_oops)
arch/x86/kernel/entry_64.S +45 -12
104 lines not on this page · raw message
include/linux/hardirq.h +11 -5
@@ -1,12 +1,14 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#ifndef __ASSEMBLY__#include <linux/preempt.h> #include <linux/smp_lock.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h> #include <asm/hardirq.h> #include <asm/system.h> +#endif /* * We put the hardirq and softirq counter into the preemption
@@ -50,17 +52,17 @@#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) -#define __IRQ_MASK (x) ((1UL << (x))-1) +#define __IRQ_MASK (x) ((1 << (x))-1)#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) -#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT) -#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) -#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) -#define NMI_OFFSET (1UL << NMI_SHIFT) +#define PREEMPT_OFFSET (1 << PREEMPT_SHIFT) +#define SOFTIRQ_OFFSET (1 << SOFTIRQ_SHIFT) +#define HARDIRQ_OFFSET (1 << HARDIRQ_SHIFT) +#define NMI_OFFSET (1 << NMI_SHIFT)#if PREEMPT_ACTIVE < (1 << (NMI_SHIFT +NMI_BITS ))#error PREEMPT_ACTIVE is too low!
@@ -116,6 +118,8 @@# define IRQ_EXIT_OFFSET HARDIRQ_OFFSET#endif +#ifndef __ASSEMBLY__ +#if defined(CONFIG_SMP ) || defined(CONFIG_GENERIC_HARDIRQS )extern void synchronize_irq (unsigned int irq);#else
@@ -195,4 +199,6 @@ extern void irq_exit(void);ftrace_nmi_exit (); \ } while (0) +#endif /* !__ASSEMBLY__ */ +#endif /* LINUX_HARDIRQ_H */
Quoted 36 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Mathieu Desnoyers
Hrm, would it be possible to save the c2 register upon nmi handler entry and restore it before iret instead ?
Yes, that would work as well, and be less subtle.
It still does have the same worries about CPU's not being all that happy about writing to %cr2 (we do it when restoring CPU state at resume time, but nobody has ever _cared_ before, so I don't know if it matters).
I think we can dodge the whole issue by asking whether the old 1-year-old patch from Mathieu (repeated below - partially ported and barely tested / not signed off) is an actual speedup in the normal exception codepaths.
The gist of it is the replacement of iret with this open-coded sequence:
arch/x86/include/asm/irqflags.h +8
-0+#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + retWhether popfq+ret is faster than iret is the question i think. (beyond the question of 'how about all the weird stack exceptions that are possible above')
Quoted 7 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
If it's slower then it's a non-starter and we do the GUP solution.
Lets hope it's slower ;-)
As I pointed out in my earlier email, and as I add :
- My patch only touched exceptions nested in NMI handlers
- You should not use the popf+ret to return to userspace. This could lead to interesting results like userspace running in ring 0. (read :
bad things would happen).
So if you want to try making it the default to return from interrupts or exceptions caught in kernel-mode, that might be interesting to see if it speeds up the system compared to iret.
Mathieu
Quoted 355 lines
Ingo
----------->
Subject: x86 NMI-safe INT3 and Page Fault
From: Mathieu Desnoyers <[email protected]>
Date: Mon, 12 May 2008 21:21:07 +0200Implements an alternative iret with popf and return so trap and exception handlers can return to the NMI handler without issuing iret. iret would cause NMIs to be reenabled prematurely.
x86_32uses popf and far return.x86_64has to copy the return instruction pointer to the top of the previous stack, issue a popf, loads the previous esp and issue a near return (ret).It allows placing immediate values (and therefore optimized
trace_marks) in NMI code since returning from a breakpoint would be valid. Accessing vmalloc'd memory, which allows executing module code or accessing vmapped or vmalloc'd areas from NMI context, would also be valid. This is very useful to tracers like LTTng.This patch makes all faults, traps and exception safe to be called from NMI context *except* single-stepping, which requires iret to restore the TF (trap flag) and jump to the return address in a single instruction. Sorry, no kprobes support in NMI handlers because of this limitation. We cannot single-step an NMI handler, because iret must set the TF flag and return back to the instruction to single-step in a single instruction. This cannot be emulated with popf/lret, because lret would be single-stepped. It does not apply to immediate values because they do not use single-stepping. This code detects if the TF flag is set and uses the iret path for single-stepping, even if it reactivates NMIs prematurely.
Test to detect if nested under a NMI handler is only done upon the return from trap/exception to kernel, which is not frequent. Other return paths (return from trap/exception to userspace, return from interrupt) keep the exact same behavior (no slowdown).
Depends on :
change-alpha-active-count-bit.patch
change-avr32-active-count-bit.patchTODO : test with lguest, xen, kvm.
** This patch depends on the "Stringify support commas" patchset ** ** Also depends on fix-x86_64-page-fault-scheduler-race patch **
tested on
x86_32(tests implemented in a separate patch) :
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.
- tested vmalloc faults in NMI handler by placing a non-optimized marker in the NMI handler (so no breakpoint is executed) and connecting a probe which touches every pages of a 20MB vmalloc'd buffer. It executes trough the return path without problem.
- Tested with and without preemptiontested on
x86_64
- instrumented the return path to export the EIP, CS and EFLAGS values when taken so we know the return path code has been executed.
-trace_mark, using immediate values, with 10ms delay with the breakpoint activated. Runs well through the return path.To test on
x86_64:
- Test without preemption
- Test vmalloc faults
- Test on Intel 64 bits CPUs. (AMD64 was fine)Changelog since v1 :
-x86_64fixes.
Changelog since v2 :
- fix paravirt build
Changelog since v3 :
- Include modifications suggested by Jeremy
Changelog since v4 :
- includinghardirq.hin entry_32/64.S is a bad idea (non ifndef'd C code), defineHARDNMI_MASKin the .S files directly.
Changelog since v5 :
- AddHARDNMI_MASKtoirq_count()and makedie()more verbose for NMIs. Changelog since v7 :
- Implement paravirtualizednmi_return.
Changelog since v8 :
- refreshed the patch for asm-offsets. Those were left out of v8.
- now depends on "Stringify support commas" patch.
Changelog since v9 :
- Only test the nmi nested preempt count flag upon return from exceptions, not on return from interrupts. Only the kernel return path has this test.
- Add Xen, VMI, lguest support. Use their iret pavavirt ops in lieu ofnmi_return.-- Ported to sched-devel.git
--- arch/x86/include/asm/irqflags.h | 56 +++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/dumpstack.c | 2 + arch/x86/kernel/entry_64.S | 57 +++++++++++++++++++++++++++++++--------- include/linux/hardirq.h | 16 +++++++----Index:
linux/arch/x86/include/asm/irqflags.h
===================================================================arch/x86/include/asm/irqflags.h +56
-0@@ -51,6 +51,61 @@ static inline void native_halt(void)#endif +#ifdef CONFIG_X86_64 +/* +* Only returns from a trap or exception to a NMI context (intra-privilege +* level near return) to the same SS and CS segments. Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. It takes care of modifying the eflags, rsp and returning to the +* previous function. +* +* The stack, at that point, looks like : +* +* 0(rsp) RIP +* 8(rsp) CS +* 16(rsp) EFLAGS +* 24(rsp) RSP +* 32(rsp) SS +* +* Upon execution : +* Copy EIP to the top of the return stack +* Update top of return stack address +* Pop eflags into the eflags register +* Make the return stack current +* Near return (popping the return address from the return stack) +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret +#else +/* +* Protected mode only, no V8086. Implies that protected mode must +* be entered before NMIs or MCEs are enabled. Only returns from a trap or +* exception to a NMI context (intra-privilege level far return). Should be used +* upon trap or exception return when nested over a NMI context so no iret is +* issued. +* +* The stack, at that point, looks like : +* +* 0(esp) EIP +* 4(esp) CS +* 8(esp) EFLAGS +* +* Upon execution : +* Copy the stack eflags to top of stack +* Pop eflags into the eflags register +* Far return: pop EIP and CS into their register, and additionally pop EFLAGS. +*/ +#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushl 8(%esp); \ + popfl; \ + lret $4 +#endif +#ifdef CONFIG_PARAVIRT #include <asm/paravirt.h> #else @@ -109,6 +164,7 @@ static inline unsigned long __raw_local_#define ENABLE_INTERRUPTS (x) sti#define DISABLE_INTERRUPTS (x) cli +#define INTERRUPT_RETURN_NMI_SAFE NATIVE_INTERRUPT_RETURN_NMI_SAFE#ifdef CONFIG_X86_64 #define SWAPGS swapgsIndex:
linux/arch/x86/kernel/dumpstack.c
===================================================================arch/x86/kernel/dumpstack.c +2
-0@@ -237,6 +237,8 @@ void __kprobes oops_end(unsigned long flif (!signr)return ; +if (in_nmi ()) +panic ("Fatal exception in non-maskable interrupt" );if (in_interrupt ())panic ("Fatal exception in interrupt" );if (panic_on_oops)Index:
linux/arch/x86/kernel/entry_64.S
===================================================================arch/x86/kernel/entry_64.S +45
-12@@ -53,6 +53,7 @@ #include <asm/paravirt.h> #include <asm/ftrace.h> #include <asm/percpu.h> +#include <linux/hardirq.h> /* Avoid __ASSEMBLER__'ifying <linux/audit.h> just for this. */ #include <linux/elf-em.h>@@ -875,6 +876,9 @@ ENTRY(native_iret) .section __ex_table,"a" .quad native_iret, bad_iret .previous + +ENTRY(native_nmi_return) + NATIVE_INTERRUPT_RETURN_NMI_SAFE #endif .section .fixup,"ax"@@ -929,6 +933,23 @@ retint_signal: GET_THREAD_INFO(%rcx) jmp retint_with_reschedule + /* Returning to kernel space from exception. */ + /* rcx: threadinfo. interrupts off. */ +ENTRY(retexc_kernel) + testl $NMI_MASK, TI_preempt_count(%rcx) + jz retint_kernel /* Not nested over NMI ? */ + testw $X86_EFLAGS_TF, EFLAGS-ARGOFFSET(%rsp) /* trap flag? */ + jnz retint_kernel /* + * If single-stepping an NMI handler, + * use the normal iret path instead of + * the popf/lret because lret would be + * single-stepped. It should not + * happen : it will reactivate NMIs + * prematurely. + */ + RESTORE_ARGS 0, 8, 0 + INTERRUPT_RETURN_NMI_SAFE + #ifdef CONFIG_PREEMPT /* Returning to kernel space. Check if we need preemption */ /* rcx: threadinfo. interrupts off. */@@ -1407,34 +1428,46 @@ ENTRY(paranoid_exit) INTR_FRAME DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF - testl %ebx,%ebx /* swapgs needed? */ + testl %ebx, %ebx /* swapgs needed? */ jnz paranoid_restore - testl $3,CS(%rsp) + + testl $3, CS(%rsp) jnz paranoid_userspace + paranoid_swapgs: TRACE_IRQS_IRETQ 0 SWAPGS_UNSAFE_STACK RESTORE_ALL 8 jmp irq_return -paranoid_restore: +paranoid_restore_no_nmi: TRACE_IRQS_IRETQ 0 RESTORE_ALL 8 jmp irq_return +paranoid_restore: + GET_THREAD_INFO(%rcx) + testl $NMI_MASK, TI_preempt_count(%rcx) + jz paranoid_restore_no_nmi /* Nested over NMI ? */ + + testw $X86_EFLAGS_TF, EFLAGS-0(%rsp) /* trap flag? */ + jnz paranoid_restore_no_nmi + RESTORE_ALL 8 + INTERRUPT_RETURN_NMI_SAFE + paranoid_userspace: GET_THREAD_INFO(%rcx) - movl TI_flags(%rcx),%ebx - andl $_TIF_WORK_MASK,%ebx + movl TI_flags(%rcx), %ebx + andl $_TIF_WORK_MASK, %ebx jz paranoid_swapgs - movq %rsp,%rdi /* &pt_regs */ + movq %rsp, %rdi /* &pt_regs */ call sync_regs - movq %rax,%rsp /* switch stack for scheduling */ - testl $_TIF_NEED_RESCHED,%ebx + movq %rax, %rsp /* switch stack for scheduling */ + testl $_TIF_NEED_RESCHED, %ebx jnz paranoid_schedule - movl %ebx,%edx /* arg3: thread flags */ + movl %ebx, %edx /* arg3: thread flags */ TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) - xorl %esi,%esi /* arg2: oldset */ - movq %rsp,%rdi /* arg1: &pt_regs */ + xorl %esi, %esi /* arg2: oldset */ + movq %rsp, %rdi /* arg1: &pt_regs */ call do_notify_resume DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF@@ -1513,7 +1546,7 @@ ENTRY(error_exit) TRACE_IRQS_OFF GET_THREAD_INFO(%rcx) testl %eax,%eax - jne retint_kernel + jne retexc_kernel LOCKDEP_SYS_EXIT_IRQ movl TI_flags(%rcx),%edx movl $_TIF_WORK_MASK,%ediIndex:
linux/include/linux/hardirq.h
===================================================================include/linux/hardirq.h +11
-5@@ -1,12 +1,14 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#ifndef __ASSEMBLY__#include <linux/preempt.h> #include <linux/smp_lock.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h> #include <asm/hardirq.h> #include <asm/system.h> +#endif /* * We put the hardirq and softirq counter into the preemption @@ -50,17 +52,17 @@#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS) -#define __IRQ_MASK (x) ((1UL << (x))-1) +#define __IRQ_MASK (x) ((1 << (x))-1)#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT) -#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT) -#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) -#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT) -#define NMI_OFFSET (1UL << NMI_SHIFT) +#define PREEMPT_OFFSET (1 << PREEMPT_SHIFT) +#define SOFTIRQ_OFFSET (1 << SOFTIRQ_SHIFT) +#define HARDIRQ_OFFSET (1 << HARDIRQ_SHIFT) +#define NMI_OFFSET (1 << NMI_SHIFT)#if PREEMPT_ACTIVE < (1 << (NMI_SHIFT +NMI_BITS ))#error PREEMPT_ACTIVE is too low!@@ -116,6 +118,8 @@# define IRQ_EXIT_OFFSET HARDIRQ_OFFSET#endif +#ifndef __ASSEMBLY__ +#if defined(CONFIG_SMP ) || defined(CONFIG_GENERIC_HARDIRQS )extern void synchronize_irq (unsigned int irq);#else @@ -195,4 +199,6 @@ extern void irq_exit(void);ftrace_nmi_exit (); \ } while (0) +#endif /* !__ASSEMBLY__ */ +#endif /* LINUX_HARDIRQ_H */
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
arch/x86/include/asm/irqflags.h static inline void native_halt(void) · line 84
Quoted 12 lines · Ingo Molnar
The gist of it is the replacement of iret with this open-coded sequence:
arch/x86/include/asm/irqflags.h +8
-0+#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + ret
That's an odd way of writing it.
Don't we have a per-cpu segment here? I'd much rather just see it do something like this (_before_ restoring the regular registers)
movq EIP(%esp),%rax movq ESP(%esp),%rdx movq %rax,gs:saved_esp movq %rdx,gs:saved_eip # restore regular regs RESTORE_ALL # skip eip/esp to get at eflags addl $16,%esp popfq # restore rsp/rip movq gs:saved_esp,%rsp jmpq *(gs:saved_eip)
but I haven't thought deeply about it. Maybe there's something wrong with the above.
Quoted 2 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
I don't think it's all that hot. It's not like it's the return to user mode.
Linus
Quoted 7 lines · * Linus Torvalds ([email protected]) wrote:
Quoted 4 lines · Ingo Molnar
The gist of it is the replacement of iret with this open-coded sequence:
Quoted 11 lines
arch/x86/include/asm/irqflags.h static inline void native_halt(void) · line 84
Quoted 8 lines
arch/x86/include/asm/irqflags.h +8
-0+#define NATIVE_INTERRUPT_RETURN_NMI_SAFE pushq %rax; \ + movq %rsp, %rax; \ + movq 24+8(%rax), %rsp; \ + pushq 0+8(%rax); \ + pushq 16+8(%rax); \ + movq (%rax), %rax; \ + popfq; \ + retThat's an odd way of writing it.
There were a few reasons (maybe not all good) for writing it like this :
- Saving I$ (as it is placed close to hot entry.S code paths)
- Staying localized with the top of stack, saving D$ accesses.
But maybe benchmarks will prove my approach overkill, dunno. Also we have to be aware that the CPU might behave more slowly in the presence of unbalanced int/iret, call/ret. I think we should benchmark your approach to make sure jmp will not produce such slowdown. But it might well be faster, and it's definitely clearer.
Thanks,
Mathieu
Quoted 29 lines
Don't we have a per-cpu segment here? I'd much rather just see it do something like this (_before_ restoring the regular registers)
movq EIP(%esp),%rax movq ESP(%esp),%rdx movq %rax,gs:saved_esp movq %rdx,gs:saved_eip # restore regular regs RESTORE_ALL # skip eip/esp to get at eflags addl $16,%esp popfq # restore rsp/rip movq gs:saved_esp,%rsp jmpq *(gs:saved_eip)but I haven't thought deeply about it. Maybe there's something wrong with the above.
Quoted 2 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
I don't think it's all that hot. It's not like it's the return to user mode.
Linus
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 5 lines · Linus Torvalds
Quoted 2 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
I don't think it's all that hot. It's not like it's the return to user mode.
Well i guess it depends. For server apps it is true - syscalls are a lot more dominant, MMs are long-running so any startup cost gets amortized and pagefaults are avoided.
For something like a kernel build we have 7 times as many pagefaults as syscalls:
aldebaran:~/linux/linux> perf stat -- make -j32 >/dev/null [...]
Performance counter stats for 'make -j32':
1444281.076741 task-clock-msecs # 14.429 CPUs
219991 context-switches # 0.000 M/sec
18335 CPU-migrations # 0.000 M/sec
38465628 page-faults # 0.027 M/sec
4374762924204 cycles # 3029.025 M/sec
2645979309823 instructions # 0.605 IPC
42398991227 cache-references # 29.356 M/sec
4371920878 cache-misses # 3.027 M/sec100.097787566 seconds time elapsed.
So we have 38465628 page-faults, or one every 68788 instructions, one every 113731 cycles.
10 cycles saved in the page fault costs means 0.01% performance win
- or about 10 milliseconds shaven off the kernel build time.
100 cycles saved (which is impossible really in the entry/exit path) would mean 0.1% win.
5653639 syscalls (according to strace -c) - which is a factor of 6.8 lower. Same goes for shell scripts or most of the clicking we do on a GUI.
It's not a big factor for sure.
Btw., the biggest pagefault cost is in the fault handling itself (the page clearing):
4.14% [k] do_page_fault
1.20% [k] sys_write
1.10% [k] sys_open
0.63% [k] sys_exit_group
0.48% [k] smp_apic_timer_interrupt
0.37% [k] sys_read
0.37% [k] sys_execve
0.20% [k] sys_mmap
0.18% [k] sys_close
0.14% [k] sys_munmap
0.13% [k] sys_poll
0.09% [k] sys_newstat
0.07% [k] sys_clone
0.06% [k] sys_newfstatit totals to 4.14% of the total cost (user-space cycles included) of a kernel build, on a Nehalem box.
Ingo
Quoted 64 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Linus Torvalds
Quoted 2 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
I don't think it's all that hot. It's not like it's the return to user mode.
Well i guess it depends. For server apps it is true - syscalls are a lot more dominant, MMs are long-running so any startup cost gets amortized and pagefaults are avoided.
For something like a kernel build we have 7 times as many pagefaults as syscalls:
aldebaran:~/linux/linux> perf stat -- make -j32 >/dev/null [...]
Performance counter stats for 'make -j32':1444281.076741 task-clock-msecs # 14.429 CPUs 219991 context-switches # 0.000 M/sec 18335 CPU-migrations # 0.000 M/sec 38465628 page-faults # 0.027 M/sec 4374762924204 cycles # 3029.025 M/sec 2645979309823 instructions # 0.605 IPC 42398991227 cache-references # 29.356 M/sec 4371920878 cache-misses # 3.027 M/sec100.097787566 seconds time elapsed.
So we have 38465628 page-faults, or one every 68788 instructions, one every 113731 cycles.
10 cycles saved in the page fault costs means 0.01% performance win
- or about 10 milliseconds shaven off the kernel build time.100 cycles saved (which is impossible really in the entry/exit path) would mean 0.1% win.
5653639 syscalls (according to strace -c) - which is a factor of 6.8 lower. Same goes for shell scripts or most of the clicking we do on a GUI.
It's not a big factor for sure.
Btw., the biggest pagefault cost is in the fault handling itself (the page clearing):
4.14% [k] do_page_fault 1.20% [k] sys_write 1.10% [k] sys_open 0.63% [k] sys_exit_group 0.48% [k] smp_apic_timer_interrupt 0.37% [k] sys_read 0.37% [k] sys_execve 0.20% [k] sys_mmap 0.18% [k] sys_close 0.14% [k] sys_munmap 0.13% [k] sys_poll 0.09% [k] sys_newstat 0.07% [k] sys_clone 0.06% [k] sys_newfstat
Quoted 3 lines
it totals to 4.14% of the total cost (user-space cycles included) of a kernel build, on a Nehalem box.
Yes, page faults caused by COW of short-lived processes and faulting-in execs and libraries account for an insane portion of the build time. :-/
Mathieu
Quoted 1 line
Ingo
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
btw., here's the cost analysis of cr2 reading and writing (in a tight loop). I've executed cr2 read+write instructions 1 billion times on a Nehalem box:
static long cr2_test(void)
{
unsigned long tmp = 0;
int i;
for (i = 0; i < 1000000000; i++)
asm("movq %0, %%cr2; movq %%cr2, %0" : : "r" (tmp));
return 0;
}Which gave these overall stats:
Performance counter stats for './prctl 0 0':
28414.696319 task-clock-msecs # 0.997 CPUs
3 context-switches # 0.000 M/sec
1 CPU-migrations # 0.000 M/sec
149 page-faults # 0.000 M/sec
87254432334 cycles # 3070.750 M/sec
5078691161 instructions # 0.058 IPC
304144 cache-references # 0.011 M/sec
28760 cache-misses # 0.001 M/sec28.501962853 seconds time elapsed.
87254432334/1000000000 ~== 87, so we have 87 cycles cost per iteration.
The annotated output shows:
aldebaran:~> perf annotate sys_prctl | grep -A 2 cr2
0.42 : ffffffff81053131: 0f 22 d1 mov %rcx,%cr2
96.56 : ffffffff81053134: 0f 20 d1 mov %cr2,%rcx
3.02 : ffffffff81053137: ff c0 inc %eax
0.00 : ffffffff81053139: 39 d0 cmp %edx,%eaxthe read/write cost ratio is 3%:96.5% (with skidding taken into account), that suggests that the reading cost of cr2 is about 2-3 cycles, the writing cost is about 85 cycles.
Which makes sense - reading cr2 is in the pagefault critical path, so that's optimized. Writing it is allowed but not optimized at all. (especially in such a tight loop where it could easily have some back-to-back additional latency that would not be there in an NMI handler save/restore path which has other instructions inbetween.)
Ingo
Quoted 13 lines · Ingo Molnar
Which gave these overall stats:
Performance counter stats for './prctl 0 0':
28414.696319 task-clock-msecs # 0.997 CPUs 3 context-switches # 0.000 M/sec 1 CPU-migrations # 0.000 M/sec 149 page-faults # 0.000 M/sec 87254432334 cycles # 3070.750 M/sec 5078691161 instructions # 0.058 IPC 304144 cache-references # 0.011 M/sec 28760 cache-misses # 0.001 M/sec
Quoted 4 lines
28.501962853 seconds time elapsed.87254432334/1000000000 ~== 87, so we have 87 cycles cost per iteration.
I also measured the GUP based copy_from_user_nmi(), on 64-bit (so there's not even any real atomic-kmap/invlpg overhead):
Performance counter stats for './prctl 0 0':
55580.513882 task-clock-msecs # 0.997 CPUs
3 context-switches # 0.000 M/sec
1 CPU-migrations # 0.000 M/sec
149 page-faults # 0.000 M/sec
176375680192 cycles # 3173.337 M/sec
299353138289 instructions # 1.697 IPC
3388060 cache-references # 0.061 M/sec
1318977 cache-misses # 0.024 M/sec55.748468367 seconds time elapsed.
This shows the overhead of looking up pagetables - 176 cycles per iteration. A cr2 save/restore pair is twice as fast.
Here's the profile btw:
aldebaran:~> perf report -s s
#
# (1813480 samples)
#
# Overhead Symbol
# ........ ......
#
23.99% [k] __get_user_pages_fast
19.89% [k] gup_pte_range
18.98% [k] gup_pud_range
16.95% [k] copy_from_user_nmi
16.04% [k] put_page
3.17% [k] sys_prctl
0.02% [k] _spin_lock
0.02% [k] copy_user_generic_string
0.02% [k] get_page_from_freelisttaking a look at 'perf annotate __get_user_pages_fast' suggests these two hot-spots:
0.04 : ffffffff810310cc: 9c pushfq
9.24 : ffffffff810310cd: 41 5d pop %r13
1.43 : ffffffff810310cf: fa cli
3.44 : ffffffff810310d0: 48 89 fb mov %rdi,%rbx
0.00 : ffffffff810310d3: 4d 8d 7e ff lea -0x1(%r14),%r15
0.00 : ffffffff810310d7: 48 c1 eb 24 shr $0x24,%rbx
0.00 : ffffffff810310db: 81 e3 f8 0f 00 00 and $0xff8,%ebx15% of its overhead is here, 50% is here:
0.71 : ffffffff81031141: 41 55 push %r13
0.05 : ffffffff81031143: 9d popfq
30.07 : ffffffff81031144: 8b 55 d4 mov -0x2c(%rbp),%edx
2.78 : ffffffff81031147: 48 83 c4 20 add $0x20,%rsp
0.00 : ffffffff8103114b: 89 d0 mov %edx,%eax
10.93 : ffffffff8103114d: 5b pop %rbx
0.02 : ffffffff8103114e: 41 5c pop %r12
1.28 : ffffffff81031150: 41 5d pop %r13
0.51 : ffffffff81031152: 41 5e pop %r14So either pushfq+cli...popfq sequences are a lot more expensive on Nehalem as i imagined, or instruction skidding is tricking us here.
gup_pte_range has a clear hotspot with a locked instruction:
2.46 : ffffffff81030d88: 48 8d 41 08 lea 0x8(%rcx),%rax
0.00 : ffffffff81030d8c: f0 ff 41 08 lock incl 0x8(%rcx)
53.52 : ffffffff81030d90: 49 63 01 movslq (%r9),%rax
0.00 : ffffffff81030d93: 48 81 c6 00 10 00 00 add $0x1000,%rsi11% of the total overhead - or about 19 cycles.
So it seems cr2+direct-access is distinctly faster than fast-gup.
And fast-gup overhead is _per frame entry_ - which makes cr2+direct-access (which is per NMI) _far_ more performant - a dozen or more call-chain entries are the norm.
Ingo
Quoted 7 lines · Ingo Molnar
Well i guess it depends. For server apps it is true - syscalls are a lot more dominant, MMs are long-running so any startup cost gets amortized and pagefaults are avoided.
For something like a kernel build we have 7 times as many pagefaults as syscalls:
Ingo - calm down.
This is not about page faults.
This is purely about taps FROM KERNEL SPACE.
Yes, for the kernel build we have 7 times as many page faults as system calls, BUT I BET 99.9% of them are from user mode!
The whole "open-code iret" only works for exceptions that happened in kernel mode. That's a _vanishingly_ small number (outside of device interrupts that happen during idle).
Linus
Quoted 15 lines · Linus Torvalds
Quoted 7 lines · Ingo Molnar
Well i guess it depends. For server apps it is true - syscalls are a lot more dominant, MMs are long-running so any startup cost gets amortized and pagefaults are avoided.
For something like a kernel build we have 7 times as many pagefaults as syscalls:
Ingo - calm down.
This is not about page faults.
Quoted 8 lines
This is purely about taps FROM KERNEL SPACE.
Yes, for the kernel build we have 7 times as many page faults as system calls, BUT I BET 99.9% of them are from user mode!
The whole "open-code iret" only works for exceptions that happened in kernel mode. That's a _vanishingly_ small number (outside of device interrupts that happen during idle).
Ah, i misunderstood you, sorry. Yes, you are right.
Btw., we can have a precise number - we can sample all pagefaults via a soft-counter:
$ perf record -e page-faults -c 1 -f -m 512 -- make -j32
[ perf record: Captured and wrote 783.697 MB perf.data (~34240256 samples) ]
#
# (33858154 samples)
#
# Overhead Symbol
# ........ ......
#
41.62% [.] memset
18.21% 0x007fff07bff634
5.85% [.] __GI_memcpy
5.35% [.] parse_config_file
4.34% [.] _int_malloc
1.24% 0x0000000000dc90
1.14% [k] file_read_actor
1.10% [.] _dl_addr
0.64% [.] __GI_strlen
0.47% [.] _dl_load_cache_lookup
0.43% [.] _dl_cache_libcmp(i dont have debuginfo installed for that gcc build, hence that raw entry.)
Kernel triggered pagefaults:
$ perf report -s s | grep '\[k\]'
1.14% [k] file_read_actor
0.15% [k] __clear_user
0.12% [k] strncpy_from_user
0.12% [k] copy_user_generic_string
0.04% [k] __put_user_4
0.04% [k] pipe_read
0.02% [k] load_elf_binary
0.00% [k] do_notify_resume
0.00% [k] iov_iter_fault_in_readableso 1.7% of the pagefaults are kernel-initiated - 98.3% via user-space. Your 99.9% figure was within 1.6% of the real number :-)
Ingo
Quoted 64 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Linus Torvalds
Quoted 2 lines
If it's faster, this becomes a legit (albeit complex) micro-optimization in a _very_ hot codepath.
I don't think it's all that hot. It's not like it's the return to user mode.
Well i guess it depends. For server apps it is true - syscalls are a lot more dominant, MMs are long-running so any startup cost gets amortized and pagefaults are avoided.
For something like a kernel build we have 7 times as many pagefaults as syscalls:
aldebaran:~/linux/linux> perf stat -- make -j32 >/dev/null [...]
Performance counter stats for 'make -j32':1444281.076741 task-clock-msecs # 14.429 CPUs 219991 context-switches # 0.000 M/sec 18335 CPU-migrations # 0.000 M/sec 38465628 page-faults # 0.027 M/sec 4374762924204 cycles # 3029.025 M/sec 2645979309823 instructions # 0.605 IPC 42398991227 cache-references # 29.356 M/sec 4371920878 cache-misses # 3.027 M/sec100.097787566 seconds time elapsed.
So we have 38465628 page-faults, or one every 68788 instructions, one every 113731 cycles.
10 cycles saved in the page fault costs means 0.01% performance win
- or about 10 milliseconds shaven off the kernel build time.100 cycles saved (which is impossible really in the entry/exit path) would mean 0.1% win.
5653639 syscalls (according to strace -c) - which is a factor of 6.8 lower. Same goes for shell scripts or most of the clicking we do on a GUI.
It's not a big factor for sure.
Btw., the biggest pagefault cost is in the fault handling itself (the page clearing):
4.14% [k] do_page_fault 1.20% [k] sys_write 1.10% [k] sys_open 0.63% [k] sys_exit_group 0.48% [k] smp_apic_timer_interrupt 0.37% [k] sys_read 0.37% [k] sys_execve 0.20% [k] sys_mmap 0.18% [k] sys_close 0.14% [k] sys_munmap 0.13% [k] sys_poll 0.09% [k] sys_newstat 0.07% [k] sys_clone 0.06% [k] sys_newfstat
Quoted 4 lines
it totals to 4.14% of the total cost (user-space cycles included) of a kernel build, on a Nehalem box.
Ingo
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
I am tempted to think that taking, say, ~10 cycles on the interrupt path worths it if we save a few tens of cycles on the page fault handler fast path.
But again, this calls for benchmarks.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 6 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
Quoted 9 lines
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
I am tempted to think that taking, say, ~10 cycles on the interrupt path worths it if we save a few tens of cycles on the page fault handler fast path.
Doesn't sound all that crazy, I suspect the underlying assumption that interrupt gates are slower than trap gates is incorrect. Disabling interrupts itself isn't expensive, it's the synchronization requirements.
-hpa
Quoted 6 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
Quoted 6 lines
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
The cost on Nehalem (1 billion local_irq_save()+restore() pairs):
aldebaran:~> perf stat --repeat 5 ./prctl 0 0
Performance counter stats for './prctl 0 0' (5 runs):
10950.813461 task-clock-msecs # 0.997 CPUs ( +- 1.594% )
3 context-switches # 0.000 M/sec ( +- 0.000% )
1 CPU-migrations # 0.000 M/sec ( +- 0.000% )
145 page-faults # 0.000 M/sec ( +- 0.000% )
33946294720 cycles # 3099.888 M/sec ( +- 1.132% )
8030365827 instructions # 0.237 IPC ( +- 0.006% )
100933 cache-references # 0.009 M/sec ( +- 12.568% )
27250 cache-misses # 0.002 M/sec ( +- 3.897% )10.985768499 seconds time elapsed.
That's 33.9 cycles per iteration, with a 1.1% confidence factor.
Annotation gives this result:
2.24 : ffffffff810535e5: 9c pushfq
8.58 : ffffffff810535e6: 58 pop %rax
10.99 : ffffffff810535e7: fa cli
20.38 : ffffffff810535e8: 50 push %rax
0.00 : ffffffff810535e9: 9d popfq
46.71 : ffffffff810535ea: ff c6 inc %esi
0.42 : ffffffff810535ec: 3b 35 72 31 76 00 cmp 0x763172(%rip),%e
10.69 : ffffffff810535f2: 7c f1 jl ffffffff810535e5
0.00 : ffffffff810535f4: e9 7c 01 00 00 jmpq ffffffff81053775 i.e. pushfq+cli is roughly 42.19% or 14 cycles, the popfq is 46.71 or 16 cycles. So the combo cost is 30 cycles, +- 1 cycle.
(Actual effective cost in a real critical section can be better than this, dependent on surrounding instructions.)
It got quite a bit faster than Core2 - but still not as fast as AMD.
Ingo
Quoted 50 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
Quoted 6 lines
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
The cost on Nehalem (1 billion local_irq_save()+restore() pairs):
aldebaran:~> perf stat --repeat 5 ./prctl 0 0
Performance counter stats for './prctl 0 0' (5 runs):
10950.813461 task-clock-msecs # 0.997 CPUs ( +- 1.594% ) 3 context-switches # 0.000 M/sec ( +- 0.000% ) 1 CPU-migrations # 0.000 M/sec ( +- 0.000% ) 145 page-faults # 0.000 M/sec ( +- 0.000% ) 33946294720 cycles # 3099.888 M/sec ( +- 1.132% ) 8030365827 instructions # 0.237 IPC ( +- 0.006% ) 100933 cache-references # 0.009 M/sec ( +- 12.568% ) 27250 cache-misses # 0.002 M/sec ( +- 3.897% )10.985768499 seconds time elapsed.That's 33.9 cycles per iteration, with a 1.1% confidence factor.
Annotation gives this result:
2.24 : ffffffff810535e5: 9c pushfq 8.58 : ffffffff810535e6: 58 pop %rax 10.99 : ffffffff810535e7: fa cli 20.38 : ffffffff810535e8: 50 push %rax 0.00 : ffffffff810535e9: 9d popfq 46.71 : ffffffff810535ea: ff c6 inc %esi 0.42 : ffffffff810535ec: 3b 35 72 31 76 00 cmp 0x763172(%rip),%e 10.69 : ffffffff810535f2: 7c f1 jl ffffffff810535e5 0.00 : ffffffff810535f4: e9 7c 01 00 00 jmpq ffffffff81053775i.e. pushfq+cli is roughly 42.19% or 14 cycles, the popfq is 46.71 or 16 cycles. So the combo cost is 30 cycles, +- 1 cycle.
Quoted 6 lines
(Actual effective cost in a real critical section can be better than this, dependent on surrounding instructions.)
It got quite a bit faster than Core2 - but still not as fast as AMD.
Ingo
Interesting, but in our specific case, what would be even more interesting to know is how many trap gates/s vs interrupt gates/s can be called. This would allow us to see if it's worth trying to make the page fault handler interrupt-safe by mean of atomicity and context save/restore by interrupt handlers (which would let us run the PF handler with interrupts enabled).
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 56 lines · Mathieu Desnoyers
Quoted 55 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
Quoted 6 lines
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
The cost on Nehalem (1 billion local_irq_save()+restore() pairs):
aldebaran:~> perf stat --repeat 5 ./prctl 0 0
Performance counter stats for './prctl 0 0' (5 runs):
10950.813461 task-clock-msecs # 0.997 CPUs ( +- 1.594% ) 3 context-switches # 0.000 M/sec ( +- 0.000% ) 1 CPU-migrations # 0.000 M/sec ( +- 0.000% ) 145 page-faults # 0.000 M/sec ( +- 0.000% ) 33946294720 cycles # 3099.888 M/sec ( +- 1.132% ) 8030365827 instructions # 0.237 IPC ( +- 0.006% ) 100933 cache-references # 0.009 M/sec ( +- 12.568% ) 27250 cache-misses # 0.002 M/sec ( +- 3.897% )10.985768499 seconds time elapsed.That's 33.9 cycles per iteration, with a 1.1% confidence factor.
Annotation gives this result:
2.24 : ffffffff810535e5: 9c pushfq 8.58 : ffffffff810535e6: 58 pop %rax 10.99 : ffffffff810535e7: fa cli 20.38 : ffffffff810535e8: 50 push %rax 0.00 : ffffffff810535e9: 9d popfq 46.71 : ffffffff810535ea: ff c6 inc %esi 0.42 : ffffffff810535ec: 3b 35 72 31 76 00 cmp 0x763172(%rip),%e 10.69 : ffffffff810535f2: 7c f1 jl ffffffff810535e5 0.00 : ffffffff810535f4: e9 7c 01 00 00 jmpq ffffffff81053775i.e. pushfq+cli is roughly 42.19% or 14 cycles, the popfq is 46.71 or 16 cycles. So the combo cost is 30 cycles, +- 1 cycle.
(Actual effective cost in a real critical section can be better than this, dependent on surrounding instructions.)
It got quite a bit faster than Core2 - but still not as fast as AMD.
Quoted 8 lines
Quoted 1 line
Ingo
Interesting, but in our specific case, what would be even more interesting to know is how many trap gates/s vs interrupt gates/s can be called. This would allow us to see if it's worth trying to make the page fault handler interrupt-safe by mean of atomicity and context save/restore by interrupt handlers (which would let us run the PF handler with interrupts enabled).
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
OTOH, interrupt gates, if above 10K/second, do get noticed and get reduced. Above 100K/sec combined they are really painful. In practice, a combo limit of 10K is healthy.
So there's about an order of magnitude difference in the frequency of IRQs versus the frequency of pagefaults.
In the worst-case, we have 10K irqs/sec and almost zero pagefaults - every 10 cycles overhead in irq entry+exit cost causes a 0.003% total slowdown.
So i'd say that it's pretty safe to say that the shuffling of overhead from the pagefault path into the irq path, even if it's a zero-sum game as per cycles, is an overall win - or even in the worst-case, a negligible overhead.
Syscalls are even more critical: it's easy to have a 'good' workload with millions of syscalls per second - so getting even a single cycle off the syscall entry+exit path is worth the pain.
Ingo
Quoted 88 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 56 lines · Mathieu Desnoyers
Quoted 55 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
Quoted 6 lines
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
The cost on Nehalem (1 billion local_irq_save()+restore() pairs):
aldebaran:~> perf stat --repeat 5 ./prctl 0 0
Performance counter stats for './prctl 0 0' (5 runs):
10950.813461 task-clock-msecs # 0.997 CPUs ( +- 1.594% ) 3 context-switches # 0.000 M/sec ( +- 0.000% ) 1 CPU-migrations # 0.000 M/sec ( +- 0.000% ) 145 page-faults # 0.000 M/sec ( +- 0.000% ) 33946294720 cycles # 3099.888 M/sec ( +- 1.132% ) 8030365827 instructions # 0.237 IPC ( +- 0.006% ) 100933 cache-references # 0.009 M/sec ( +- 12.568% ) 27250 cache-misses # 0.002 M/sec ( +- 3.897% )10.985768499 seconds time elapsed.That's 33.9 cycles per iteration, with a 1.1% confidence factor.
Annotation gives this result:
2.24 : ffffffff810535e5: 9c pushfq 8.58 : ffffffff810535e6: 58 pop %rax 10.99 : ffffffff810535e7: fa cli 20.38 : ffffffff810535e8: 50 push %rax 0.00 : ffffffff810535e9: 9d popfq 46.71 : ffffffff810535ea: ff c6 inc %esi 0.42 : ffffffff810535ec: 3b 35 72 31 76 00 cmp 0x763172(%rip),%e 10.69 : ffffffff810535f2: 7c f1 jl ffffffff810535e5 0.00 : ffffffff810535f4: e9 7c 01 00 00 jmpq ffffffff81053775i.e. pushfq+cli is roughly 42.19% or 14 cycles, the popfq is 46.71 or 16 cycles. So the combo cost is 30 cycles, +- 1 cycle.
(Actual effective cost in a real critical section can be better than this, dependent on surrounding instructions.)
It got quite a bit faster than Core2 - but still not as fast as AMD.
Quoted 8 lines
Quoted 1 line
Ingo
Interesting, but in our specific case, what would be even more interesting to know is how many trap gates/s vs interrupt gates/s can be called. This would allow us to see if it's worth trying to make the page fault handler interrupt-safe by mean of atomicity and context save/restore by interrupt handlers (which would let us run the PF handler with interrupts enabled).
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
OTOH, interrupt gates, if above 10K/second, do get noticed and get reduced. Above 100K/sec combined they are really painful. In practice, a combo limit of 10K is healthy.
So there's about an order of magnitude difference in the frequency of IRQs versus the frequency of pagefaults.
In the worst-case, we have 10K irqs/sec and almost zero pagefaults - every 10 cycles overhead in irq entry+exit cost causes a 0.003% total slowdown.
So i'd say that it's pretty safe to say that the shuffling of overhead from the pagefault path into the irq path, even if it's a zero-sum game as per cycles, is an overall win - or even in the worst-case, a negligible overhead.
Quoted 5 lines
Syscalls are even more critical: it's easy to have a 'good' workload with millions of syscalls per second - so getting even a single cycle off the syscall entry+exit path is worth the pain.
Ingo
I fully agree with what you say here Ingo, but then I think I must make my main point a bit more clear :
Trap handlers are currently defined as "interrupt gates" rather than trap gates, so interrupts are disabled starting from the moment the page fault is generated. This is done, as Linus said, to protect the content of the cr2 register from being messed up by interrupts. However, if we choose to save the cr2 register around irq handler execution, we could turn the page fault handler into a "real" trap gate (with interrupts on).
Given I think, just like you, that we must save cycles on the PF handler path, it would be interesting to see if there is a performance gain to get by switching the pf handler from interrupt gate to trap gate.
So the test would be :
traps.c: set_intr_gate(14, &page_fault);
changed for something like a set_trap_gate.
But we should make sure to save the cr2 register upon interrupt/NMI entry and restore it upon int/NMI exit.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 4 lines · Ingo Molnar
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
Did you do any function-level profiles?
Last I looked at it, the real cost of page faults were all in the memory copies and page clearing, and while it would be nice to speed up the kernel entry and exit, the few tens of cycles we might be able to get from there really aren't all that important.
38 million page faults may sound like a lot, but put it in other terms: if we get rid of 20 cycles for each page fault, that's still not a lot of actual time. Lookie here at your own numbers:
38465628 page-faults # 0.027 M/sec 4374762924204 cycles # 3029.025 M/sec
Now, if we shave 20 cycles off each page fault, that is still just, what, 0.018% or something? Not really that impressive in the end.
So I'm all for optimizing the kernel entry/exit paths, but ..
Linus
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
Quoted 6 lines
Did you do any function-level profiles?
Last I looked at it, the real cost of page faults were all in the memory copies and page clearing, and while it would be nice to speed up the kernel entry and exit, the few tens of cycles we might be able to get from there really aren't all that important.
Yeah.
Here's the function level profiles of a typical kernel build on a Nehalem box:
$ perf report --sort symbol
#
# (14317328 samples)
#
# Overhead Symbol
# ........ ......
#
44.05% 0x000000001a0b80
5.09% 0x0000000001d298
3.56% 0x0000000005742c
2.48% 0x0000000014026d
2.31% 0x00000000007b1a
2.06% 0x00000000115ac9
1.83% [.] _int_malloc
1.71% 0x00000000064680
1.50% [.] memset
1.37% 0x00000000125d88
1.28% 0x000000000b7642
1.17% [k] clear_page_c
0.87% [k] page_fault
0.78% [.] is_defined_config
0.71% [.] _int_free
0.68% [.] __GI_strlen
0.66% 0x000000000699e8
0.54% [.] __GI_memcpyMost is dominated by user-space symbols. (no proper ELF+debuginfo on this box so they are unnamed.) It also sows that page clearing and pagefault handling dominates the kernel overhead - but is dwarved by other overhead. Any page-fault-entry costs are a drop in the bucket.
In fact with call-chain graphs we can get a precise picture, as we can do a non-linear 'slice' set operation over the samples and filter out the ones that have the 'page_fault' pattern in one of their parent functions:
$ perf report --sort symbol --parent page_fault
#
# (14317328 samples)
#
# Overhead Symbol
# ........ ......
#
1.12% [k] clear_page_c
0.87% [k] page_fault
0.43% [k] get_page_from_freelist
0.25% [k] _spin_lock
0.24% [k] do_page_fault
0.23% [k] perf_swcounter_ctx_event
0.16% [k] perf_swcounter_event
0.15% [k] handle_mm_fault
0.15% [k] __alloc_pages_nodemask
0.14% [k] __rmqueue
0.12% [k] find_get_page
0.11% [k] copy_page_c
0.11% [k] find_vma
0.10% [k] _spin_lock_irqsave
0.10% [k] __wake_up_bit
0.09% [k] _spin_unlock_irqrestore
0.09% [k] do_anonymous_page
0.09% [k] __inc_zone_stateThis "sub-profile" shows the true summary overhead that 'page_fault' and all its child functions have. Note that for example clear_page_c decreased from 1.17% to 1.12%:
1.12% [k] clear_page_c
1.17% [k] clear_page_cbecause there's 0.05% of other callers to clear_page_c() that do not involve page_fault. Those are filtered out via --parent filtering/matching.
Ingo
Quoted 23 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
Quoted 6 lines
Did you do any function-level profiles?
Last I looked at it, the real cost of page faults were all in the memory copies and page clearing, and while it would be nice to speed up the kernel entry and exit, the few tens of cycles we might be able to get from there really aren't all that important.
Yeah.
Here's the function level profiles of a typical kernel build on a Nehalem box:
$ perf report --sort symbol
Quoted 7 lines
#
# (14317328 samples)
#
# Overhead Symbol# ........ ...... # 44.05% 0x000000001a0b80
It makes me wonder how the following scenario is accounted :
- Execution of a newly forked/exec'd process instruction causes a fault. (traps, faults and interrupts can take roughly 2000 cycles to execute)
- PC sampling interrupt fires.
Will it account the execution time as part of user-space or kernel-space execution ?
Depending on how the sampling mechanism finds out if it is running in kernel mode or userspace mode, this might make the userspace PC appear as currently running even though the current execution context is the very beginning of the page fault handler (1st instruction servicing the fault).
Mathieu
Quoted 67 lines
5.09% 0x0000000001d298 3.56% 0x0000000005742c 2.48% 0x0000000014026d 2.31% 0x00000000007b1a 2.06% 0x00000000115ac9 1.83% [.] _int_malloc 1.71% 0x00000000064680 1.50% [.] memset 1.37% 0x00000000125d88 1.28% 0x000000000b7642 1.17% [k] clear_page_c 0.87% [k] page_fault 0.78% [.] is_defined_config 0.71% [.] _int_free 0.68% [.] __GI_strlen 0.66% 0x000000000699e8 0.54% [.] __GI_memcpyMost is dominated by user-space symbols. (no proper ELF+debuginfo on this box so they are unnamed.) It also sows that page clearing and pagefault handling dominates the kernel overhead - but is dwarved by other overhead. Any page-fault-entry costs are a drop in the bucket.
In fact with call-chain graphs we can get a precise picture, as we can do a non-linear 'slice' set operation over the samples and filter out the ones that have the '
page_fault' pattern in one of their parent functions:$ perf report --sort symbol --parent page_fault#
# (14317328 samples)
#
# Overhead Symbol# ........ ...... # 1.12% [k] clear_page_c 0.87% [k] page_fault 0.43% [k] get_page_from_freelist 0.25% [k] _spin_lock 0.24% [k] do_page_fault 0.23% [k] perf_swcounter_ctx_event 0.16% [k] perf_swcounter_event 0.15% [k] handle_mm_fault 0.15% [k] __alloc_pages_nodemask 0.14% [k] __rmqueue 0.12% [k] find_get_page 0.11% [k] copy_page_c 0.11% [k] find_vma 0.10% [k] _spin_lock_irqsave 0.10% [k] __wake_up_bit 0.09% [k] _spin_unlock_irqrestore 0.09% [k] do_anonymous_page 0.09% [k] __inc_zone_stateThis "sub-profile" shows the true summary overhead that '
page_fault' and all its child functions have. Note that for exampleclear_page_cdecreased from 1.17% to 1.12%:1.12% [k] clear_page_c 1.17% [k] clear_page_cbecause there's 0.05% of other callers to
clear_page_c()that do not involvepage_fault. Those are filtered out via --parent filtering/matching.Ingo
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 34 lines · Mathieu Desnoyers
Quoted 23 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar
See the numbers in the other mail: about 33 million pagefaults happen in a typical kernel build - that's ~400K/sec - and that is not a particularly really pagefault-heavy workload.
Quoted 6 lines
Did you do any function-level profiles?
Last I looked at it, the real cost of page faults were all in the memory copies and page clearing, and while it would be nice to speed up the kernel entry and exit, the few tens of cycles we might be able to get from there really aren't all that important.
Yeah.
Here's the function level profiles of a typical kernel build on a Nehalem box:
$ perf report --sort symbolQuoted 7 lines
#
# (14317328 samples)
#
# Overhead Symbol# ........ ...... # 44.05% 0x000000001a0b80It makes me wonder how the following scenario is accounted :
Quoted 8 lines
- Execution of a newly forked/exec'd process instruction causes a fault. (traps, faults and interrupts can take roughly 2000 cycles to execute)
- PC sampling interrupt fires.
Will it account the execution time as part of user-space or kernel-space execution ?
"It depends".
With call-chain profiling ("perf record --call-graph" + "perf report --sort parent") this will show up as:
#
# (19525018 samples)
#
# Overhead Parent symbol
# ........ ....................
#
88.60% [other]
4.96% do_page_fault
1.74% sys_write
1.18% sys_openat
0.75% sys_exit_group
0.74% sys_execve
0.43% sys_read
0.28% sys_mmap
0.23% sys_clone
0.21% sys_close
0.17% sys_munmap
0.15% sys_poll
0.13% sys_lstat
0.09% sys_faccessat
0.05% sys_mprotect This line:
4.96% do_page_fault
Is the summed up overhead of all things page faults.
If you sort by a specific user-space symbol, then _its_ own generated page-faults will be displayed.
Say, you profile 'git gc' done in Git's repo with 10 KHz:
perf record -g -f -F 10000 -- ./git gc
Raw outline of overhead categories:
$ perf report --sort parent
#
# Overhead Parent symbol
# ........ ....................
#
96.97% [other]
1.32% do_page_fault
0.54% sys_write
0.21% sys_exit_group
0.15% sys_open
0.14% sys_execve
0.13% sys_mmap
0.11% sys_poll
0.10% sys_clone Note that do_page_fault has 1.32% total overhead there. But if you only look at main's overhead:
#
# Overhead Symbol
# ........ ......
#
33.12% [.] lookup_object
11.17% [.] __GI_strlen
5.14% [.] decode_tree_entry
2.94% [.] __GI_memcpy
2.58% [.] find_pack_entry_one
2.30% [.] lookup_blob
1.61% [.] tree_entry
1.16% [.] process_tree
....
0.08% [k] page_fault
0.02% [k] do_page_fault
0.02% [k] page_fault
0.02% [k] filemap_fault
0.02% [k] __do_fault
0.01% [k] handle_mm_faultThe page fault overhead is down the bottom. Why? Because most pagefaults are not raised by 'main', but by the dynamic loader which runs sooner than that.
Quoted 5 lines
Depending on how the sampling mechanism finds out if it is running in kernel mode or userspace mode, this might make the userspace PC appear as currently running even though the current execution context is the very beginning of the page fault handler (1st instruction servicing the fault).
It's much more nuanced than a binary 'user-space' versus 'kernel-space' decision.
A true 'raw' call-chain looks like this:
0x25b0 [0x108]: PERF_EVENT (IP, 5): 3455: 0xffffffff810b63ad period: 310083 ... chain: nr:28
..... 0: ffffffffffffff80 ..... 1: ffffffff810b63ad ..... 2: ffffffff81018258 ..... 3: ffffffff810aeddb ..... 4: ffffffff810af14d ..... 5: ffffffff81019042 ..... 6: ffffffff8153245e ..... 7: ffffffff81533783 ..... 8: ffffffff815337cd ..... 9: ffffffff8105fc8c
..... 10: ffffffff81531c2a
..... 11: ffffffff81531e0e
..... 12: ffffffff8153174a
..... 13: ffffffff810b68aa
..... 14: ffffffff810daa24
..... 15: ffffffff810c558e
..... 16: ffffffff810c78e9
..... 17: ffffffff81533739
..... 18: ffffffff815314ff
..... 19: ffffffff810b1716
..... 20: ffffffff810b22a2
..... 21: ffffffff810e5586
..... 22: ffffffff810e6080
..... 23: ffffffff810e61a8
..... 24: ffffffff8100bd9b
..... 25: fffffffffffffe00
..... 26: 0000003641ed6590
..... 27: 0000003646e046b3
... thread: git:3455
...... dso: [kernel]
25 kernel-context RIPs followed by a context separator (fffffffffffffe00) followed by two user-space RIPs.
So whether this is kernel-space or user-space sample depends on the analysis stage - how you decide to look at it via perf report. If you only look at the top surface via 'perf report --sort symbol' it's a "kernel-space" sample. If you look deeper, it could be a user-space one too.
The full list of contexts is:
enum perf_callchain_context {
PERF_CONTEXT_HV = (__u64)-32,
PERF_CONTEXT_KERNEL = (__u64)-128,
PERF_CONTEXT_USER = (__u64)-512,
PERF_CONTEXT_GUEST = (__u64)-2048,
PERF_CONTEXT_GUEST_KERNEL = (__u64)-2176,
PERF_CONTEXT_GUEST_USER = (__u64)-2560,
PERF_CONTEXT_MAX = (__u64)-4095,
};and a call-chain can in theory include all of these, in a nice stack of call-chain entries.
( Btw., we are planning to adding context separators for IRQ and
softirq contexts as well - to be able to isolate hardirq and softirq workloads (separated away from the mostly unrelated syscall level and user-level execution overhead). )
Hope this answers your questions,
Ingo
Quoted 13 lines · Mathieu Desnoyers
In the category "crazy ideas one should never express out loud", I could add the following. We could choose to save/restore the cr2 register on the local stack at every interrupt entry/exit, and therefore allow the page fault handler to execute with interrupts enabled.
I have not benchmarked the interrupt disabling overhead of the page fault handler handled by starting an interrupt-gated handler rather than trap-gated handler, but cli/sti instructions are known to take quite a few cycles on some architectures. e.g. 131 cycles for the pair on P4, 23 cycles on AMD Athlon X2 64, 43 cycles on Intel Core2.
Quoted 5 lines
I am tempted to think that taking, say, ~10 cycles on the interrupt path worths it if we save a few tens of cycles on the page fault handler fast path.
But again, this calls for benchmarks.
One absolutely non-trivial complication with such a scheme would be
preemptability: if we enter #PF with irqs enabled then it's immediately preemptible on CONFIG_PREEMPT=y. The scheduler would switch away to another context and the cr2 value is lost before it has been read out.
This means an additional collateral damage to context-switch cr2. (which might still be worth it given that context-switches are a less hot codepath than pagefaults - but an additional complicaton.)
The ideal solution would be for the CPU to atomically push the cr2 value to the #PF hardware stack, alongside the error code it already pushes there.
Ingo
H. Peter Anvin
reviewerQuoted 5 lines · Ingo Molnar
The ideal solution would be for the CPU to atomically push the cr2 value to the #PF hardware stack, alongside the error code it already pushes there.
That's not going to happen any time soon, obviously.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 21 lines · Linus Torvalds
That's an odd way of writing it.
Don't we have a per-cpu segment here? I'd much rather just see it do something like this (_before_ restoring the regular registers)
movq EIP(%esp),%rax movq ESP(%esp),%rdx movq %rax,gs:saved_esp movq %rdx,gs:saved_eip # restore regular regs RESTORE_ALL # skip eip/esp to get at eflags addl $16,%esp popfq # restore rsp/rip movq gs:saved_esp,%rsp jmpq *(gs:saved_eip)
Quoted 3 lines
but I haven't thought deeply about it. Maybe there's something wrong with the above.
We have to restore the usermode %gs somewhere...
J
Quoted 2 lines · Jeremy Fitzhardinge
We have to restore the usermode %gs somewhere...
None of this is useful for user-mode return _anyway_, since you have to restore %cs/%ss too. At that point, you have to use iret.
Linus
H. Peter Anvin
reviewerQuoted 7 lines · Linus Torvalds
Quoted 1 line · Jeremy Fitzhardinge
We have to restore the usermode %gs somewhere...
None of this is useful for user-mode return _anyway_, since you have to restore %cs/%ss too. At that point, you have to use iret.
cs/ss you could potentially restore with sysret/sysexit, at least for the common case. Of course, this means more cases...
-hpa
Quoted 10 lines · H. Peter Anvin
Quoted 6 lines · Linus Torvalds
Quoted 1 line · Jeremy Fitzhardinge
We have to restore the usermode %gs somewhere...
None of this is useful for user-mode return _anyway_, since you have to restore %cs/%ss too. At that point, you have to use iret.
cs/ss you could potentially restore with sysret/sysexit, at least for the common case. Of course, this means more cases...
hm, does this really work? Using sysret there would be quite tempting. Does anyone know the rough cycle count difference between IRET and SYSRET on contemporary hardware?
Also, is SYSRET NMI-invariant? If yes then this would be a quite clean all-around solution: on modern hw we'd standardize on doing SYSRET from pretty much all the contexts. We'd get a nice speedup and also the NMI nested pagefaults fix.
Oh, compat mode. Doesnt SYSRET on Intel CPUs have the problem of not being able to switch back to 32-bit user-space?
Ingo
H. Peter Anvin
reviewerQuoted 5 lines · Ingo Molnar
hm, does this really work? Using sysret there would be quite tempting. Does anyone know the rough cycle count difference between IRET and SYSRET on contemporary hardware?
Quoted 8 lines
Also, is SYSRET NMI-invariant? If yes then this would be a quite clean all-around solution: on modern hw we'd standardize on doing SYSRET from pretty much all the contexts. We'd get a nice speedup and also the NMI nested pagefaults fix.
Oh, compat mode. Doesnt SYSRET on Intel CPUs have the problem of not being able to switch back to 32-bit user-space?
Not sure. SYSRET/SYSEXIT are *not* general return to userspace solutions in either case; any kind of complex modes and they can't.
And they are, of course, only applicable for returning to userspace. As such, I don't understand the "NMI invariant" comment.
-hpa
Quoted 15 lines · H. Peter Anvin
Quoted 5 lines · Ingo Molnar
hm, does this really work? Using sysret there would be quite tempting. Does anyone know the rough cycle count difference between IRET and SYSRET on contemporary hardware?
Quoted 8 lines
Also, is SYSRET NMI-invariant? If yes then this would be a quite clean all-around solution: on modern hw we'd standardize on doing SYSRET from pretty much all the contexts. We'd get a nice speedup and also the NMI nested pagefaults fix.
Oh, compat mode. Doesnt SYSRET on Intel CPUs have the problem of not being able to switch back to 32-bit user-space?
Quoted 7 lines
Not sure. SYSRET/SYSEXIT are *not* general return to userspace solutions in either case; any kind of complex modes and they can't.
And they are, of course, only applicable for returning to userspace. As such, I don't understand the "NMI invariant" comment.
Yeah - it makes no sense for the NMI return indeed, as the target CS/SS is hardcoded indeed.
Ingo
Quoted 3 lines · H. Peter Anvin
cs/ss you could potentially restore with sysret/sysexit, at least for the common case. Of course, this means more cases...
Yes. But it gets pretty complicated. You now have to make user space restore some of the registers, so you end up having to write to the user space stack etc.
It might be worth it (iret really is very slow, _especially_ to user space, but it's definitely nontrivial.
Linus
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Jeremy Fitzhardinge
We have to restore the usermode %gs somewhere...
None of this is useful for user-mode return _anyway_, since you have to restore %cs/%ss too. At that point, you have to use iret.
Ah, right. I'd even looked at that at one point, when I was investigating optimising kernel->kernel exit paths (I'd decided it wasn't worth it for its own sake, since they simply don't happen that often).
J
H. Peter Anvin
reviewerQuoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar
A simple cr2 corruption would explain all those cc1 SIGSEGVs and other user-space crashes i saw, with sufficiently intense sampling - easily.
Quoted 8 lines
Note that we could work around the %cr2 issue, since any corruption is always nicely "nested" (ie there are never any SMP issues with async writes to the register).
So what we _could_ do is to have a magic value for %cr2, along with a "NMI sequence count", and if we see that value, we just return (without doing anything) from the page fault handler.
Wouldn't it be simpler to just require the NMI handler to save and restore %cr2 around any potentially faulting references?
-hpa
Quoted 16 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 6 lines · Linus Torvalds
Quoted 3 lines · Ingo Molnar
A simple cr2 corruption would explain all those cc1 SIGSEGVs and other user-space crashes i saw, with sufficiently intense sampling - easily.
Quoted 8 lines
Note that we could work around the %cr2 issue, since any corruption is always nicely "nested" (ie there are never any SMP issues with async writes to the register).
So what we _could_ do is to have a magic value for %cr2, along with a "NMI sequence count", and if we see that value, we just return (without doing anything) from the page fault handler.
Quoted 4 lines
Wouldn't it be simpler to just require the NMI handler to save and restore %cr2 around any potentially faulting references?
-hpa
If we require that around the whole NMI handler execution, then we get all vmalloc + module text code references handled for free. This would be a nice-to-have.a And given nmi-handler is not such a frequent code path, we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
But we do, perf counters very much cares about nmi performance.
Quoted 6 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
But we do, perf counters very much cares about nmi performance.
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 5 lines · Mathieu Desnoyers
Quoted 4 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
Quoted 8 lines
Quoted 2 lines
But we do, perf counters very much cares about nmi performance.
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
Ah, well, I have no idea who expensive cr2 is, if its like a regular register then it should be fine. If however its tons more expensive then we should really avoid it.
As to the freq, 100kHz would be nice ;-)
H. Peter Anvin
reviewerQuoted 13 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Quoted 5 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
But we do, perf counters very much cares about nmi performance.
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
Quoted 6 lines
Ah, well, I have no idea who expensive cr2 is, if its like a regular register then it should be fine. If however its tons more expensive then we should really avoid it.
As to the freq, 100kHz would be nice ;-)
Writing control registers is serializing, so it's a lot more expensive than writing a normal register; my *guess* is that it will be on the order of 100-200 cycles.
That is not based on any actual information.
-hpa
Quoted 18 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 17 lines · Peter Zijlstra
Quoted 11 lines · Mathieu Desnoyers
Quoted 5 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
But we do, perf counters very much cares about nmi performance.
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
Ah, well, I have no idea who expensive cr2 is, if its like a regular register then it should be fine. If however its tons more expensive then we should really avoid it.
Quoted 9 lines
Quoted 2 lines
As to the freq, 100kHz would be nice ;-)
Writing control registers is serializing, so it's a lot more expensive than writing a normal register; my *guess* is that it will be on the order of 100-200 cycles.
That is not based on any actual information.
Then how about just writing to the cr2 register *if* it has changed while the NMI handler was running ?
if (unlikely(read_cr2() != saved_cr2)))
write_cr2(saved_cr2)
Mathieu
Quoted 2 lines
-hpa
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 8 lines · Mathieu Desnoyers
Quoted 7 lines
Quoted 1 line
Writing control registers is serializing, so it's a lot more expensive than writing a normal register; my *guess* is that it will be on the order of 100-200 cycles.
That is not based on any actual information.
Quoted 8 lines
Then how about just writing to the cr2 register *if* it has changed while the NMI handler was running ?
if (
unlikely(read_cr2() != saved_cr2)))write_cr2(saved_cr2)Mathieu
That works fine, obviously, and although it's probably overkill it's also a trivially cheap optimization.
-hpa
Quoted 12 lines · H. Peter Anvin
Quoted 11 lines · Mathieu Desnoyers
Quoted 7 lines
Quoted 1 line
Writing control registers is serializing, so it's a lot more expensive than writing a normal register; my *guess* is that it will be on the order of 100-200 cycles.
That is not based on any actual information.
Then how about just writing to the cr2 register *if* it has changed while the NMI handler was running ?
Quoted 8 lines
Quoted 5 lines
if (
unlikely(read_cr2() != saved_cr2)))write_cr2(saved_cr2)Mathieu
That works fine, obviously, and although it's probably overkill it's also a trivially cheap optimization.
Writing cr2 costs 84 cycles so it's not overkill at all - it's a nice optimization!
Btw., we dont have to re-read it - we actually _know_ when we got a fault (the fault handler gives us back an error code).
So we can do this common optimization and avoid the cr2 write usually. We only need the cr2 read.
Hm, tempting ...
Ingo
Quoted 21 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 18 lines · H. Peter Anvin
Quoted 16 lines · Mathieu Desnoyers
Quoted 7 lines
Quoted 1 line
Writing control registers is serializing, so it's a lot more expensive than writing a normal register; my *guess* is that it will be on the order of 100-200 cycles.
That is not based on any actual information.
Then how about just writing to the cr2 register *if* it has changed while the NMI handler was running ?
if (
unlikely(read_cr2() != saved_cr2)))write_cr2(saved_cr2)Mathieu
Quoted 9 lines
Quoted 2 lines
That works fine, obviously, and although it's probably overkill it's also a trivially cheap optimization.
Writing cr2 costs 84 cycles so it's not overkill at all - it's a nice optimization!
Btw., we dont have to re-read it - we actually _know_ when we got a fault (the fault handler gives us back an error code).
Just for the sake of making NMI handlers less tricky, supporting page faults caused by faulting kernel instructions (rather than only supporting explicit faulting from get_user_pages_inatomic) would be rather nice design-wise if it only costs 2-3 cycles.
And I would not want to touch the page fault handler itself to write the saved cr2 value before the handler exits, because this would add a branch on a very hot path.
Mathieu
Quoted 6 lines
So we can do this common optimization and avoid the cr2 write usually. We only need the cr2 read.
Hm, tempting ...
Ingo
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 8 lines · Mathieu Desnoyers
Just for the sake of making NMI handlers less tricky, supporting page faults caused by faulting kernel instructions (rather than only supporting explicit faulting from
get_user_pages_inatomic) would be rather nice design-wise if it only costs 2-3 cycles.And I would not want to touch the page fault handler itself to write the saved cr2 value before the handler exits, because this would add a branch on a very hot path.
_That_ path is not hot at all - it's the 'we are in atomic section and faulted' rare path (laced with an exception table search - which is extremely slow compared to other bits of the pagefault path).
But ... it's not an issue: a check can be made in the NMI code too, as we always know about pagefaults there, by virtue of getting -EFAULT back from the attempted-user-copy.
Ingo
Quoted 12 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 8 lines · Mathieu Desnoyers
Just for the sake of making NMI handlers less tricky, supporting page faults caused by faulting kernel instructions (rather than only supporting explicit faulting from
get_user_pages_inatomic) would be rather nice design-wise if it only costs 2-3 cycles.And I would not want to touch the page fault handler itself to write the saved cr2 value before the handler exits, because this would add a branch on a very hot path.
Quoted 7 lines
_That_ path is not hot at all - it's the 'we are in atomic section and faulted' rare path (laced with an exception table search - which is extremely slow compared to other bits of the pagefault path).
But ... it's not an issue: a check can be made in the NMI code too, as we always know about pagefaults there, by virtue of getting
-EFAULTback from the attempted-user-copy.
As the maintainer of the out-of-tree LTTng tracer, which hooks in the page fault handler with tracepoints, and which can build almost entirely as modules, I am very tempted to argue that having the nmi-code entirely robust wrt in-kernel page faults would be a very-nice-to-have feature.
Requiring that code to be either built-in or to call vmalloc_sync_all() after any vmalloc or after module load is just painful and error-prone. Plus, tracing is a feature that some users will only use in specific occasions. I don't see why it should be built-into the kernel at all. That's just a waste of memory in many cases. (I am not talking about users who want to do continuous system tracing here, which is a totally different scenario).
Mathieu
Quoted 2 lines
Ingo
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 6 lines · Mathieu Desnoyers
As the maintainer of the out-of-tree LTTng tracer, which hooks in the page fault handler with tracepoints, and which can build almost entirely as modules, I am very tempted to argue that having the nmi-code entirely robust wrt in-kernel page faults would be a very-nice-to-have feature.
I doubt that is ever going to be reliable, due to reentrancy issues.
-hpa
Quoted 11 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
As the maintainer of the out-of-tree LTTng tracer, which hooks in the page fault handler with tracepoints, and which can build almost entirely as modules, I am very tempted to argue that having the nmi-code entirely robust wrt in-kernel page faults would be a very-nice-to-have feature.
I doubt that is ever going to be reliable, due to reentrancy issues.
-hpa
Do you mean the page fault handler code is no ever going to be reliable or the tracer code ?
I spent a great deal of effort making LTTng lockless and reentrant wrt NMIs. It would be great if the low-level kernel exception handlers would do the same, therefore I would not have to isolate the tracer from the kernel as I currently do. Well, I would still continue to isolate the tracer from the kernel, but at least I would not have to spend as much effort controlling what exceptions and faults paths the tracer is executing.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 14 lines · Mathieu Desnoyers
Quoted 9 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
As the maintainer of the out-of-tree LTTng tracer, which hooks in the page fault handler with tracepoints, and which can build almost entirely as modules, I am very tempted to argue that having the nmi-code entirely robust wrt in-kernel page faults would be a very-nice-to-have feature.
I doubt that is ever going to be reliable, due to reentrancy issues.
-hpa
Do you mean the page fault handler code is no ever going to be reliable or the tracer code ?
Quoted 8 lines
I spent a great deal of effort making LTTng lockless and reentrant wrt NMIs. It would be great if the low-level kernel exception handlers would do the same, therefore I would not have to isolate the tracer from the kernel as I currently do. Well, I would still continue to isolate the tracer from the kernel, but at least I would not have to spend as much effort controlling what exceptions and faults paths the tracer is executing.
So instead you want to core kernel to do your work for you?
-hpa
Quoted 24 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 14 lines · Mathieu Desnoyers
Quoted 9 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
As the maintainer of the out-of-tree LTTng tracer, which hooks in the page fault handler with tracepoints, and which can build almost entirely as modules, I am very tempted to argue that having the nmi-code entirely robust wrt in-kernel page faults would be a very-nice-to-have feature.
I doubt that is ever going to be reliable, due to reentrancy issues.
-hpa
Do you mean the page fault handler code is no ever going to be reliable or the tracer code ?
Quoted 8 lines
I spent a great deal of effort making LTTng lockless and reentrant wrt NMIs. It would be great if the low-level kernel exception handlers would do the same, therefore I would not have to isolate the tracer from the kernel as I currently do. Well, I would still continue to isolate the tracer from the kernel, but at least I would not have to spend as much effort controlling what exceptions and faults paths the tracer is executing.
Quoted 2 lines
So instead you want to core kernel to do your work for you?
I'm just thinking that touching or not vmalloc'd areas should not be the first thing that comes into the mind of someone willing to write a nmi-hooking tracer or oprofile module.
As far as just me and LTTng are concerned, it's already dealt with. I'm just saying that having to deal with that at the tracer or profiler module level is ugly.
So basically, in order to let tracer/profiler modules be more solid, yes, I think the core kernel should do that work if it does not involve any significant performance degradation. Especially due to the hard-to-reproduce and hard-to-debug nature of problems caused by such races.
Mathieu
Quoted 2 lines
-hpa
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 5 lines · Mathieu Desnoyers
I'm just thinking that touching or not vmalloc'd areas should not be the first thing that comes into the mind of someone willing to write a nmi-hooking tracer or oprofile module.
But is HAS to be. Otherwise you're in deadlock city anyway.
-hpa
Quoted 9 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
I'm just thinking that touching or not vmalloc'd areas should not be the first thing that comes into the mind of someone willing to write a nmi-hooking tracer or oprofile module.
But is HAS to be. Otherwise you're in deadlock city anyway.
Where is the kernel page fault handler grabbing any lock to service in-kernel page faults exactly ?
Those are usually considered as utterly simple page table fixups, nothing more.
Mathieu
Quoted 2 lines
-hpa
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 7 lines · Mathieu Desnoyers
Where is the kernel page fault handler grabbing any lock to service in-kernel page faults exactly ?
Those are usually considered as utterly simple page table fixups, nothing more.
It doesn't, *because it doesn't have to*. Your proposal requires that page faults can be handled inside the page fault handler, and that's a pretty tall order.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 9 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 7 lines · Mathieu Desnoyers
Where is the kernel page fault handler grabbing any lock to service in-kernel page faults exactly ?
Those are usually considered as utterly simple page table fixups, nothing more.
Quoted 6 lines
It doesn't, *because it doesn't have to*. Your proposal requires that page faults can be handled inside the page fault handler, and that's a pretty tall order.
-hpa
I am not asking for the pf handler to handle every possible kind of fault recursively. Just to keep the in-kernel page fault related code for vmalloc (and possibly for prefetch ?) paths NMI-reentrant :
void do_page_fault(struct pt_regs *regs, unsigned long error_code)
address = read_cr2();
if (unlikely(kmmio_fault(regs, address)))
return;
#ifdef CONFIG_X86_32
if (unlikely(address >= TASK_SIZE)) {
#else
if (unlikely(address >= TASK_SIZE64)) {
#endif
if (!(error_code & (PF_RSVD|PF_USER|PF_PROT)) &&
vmalloc_fault(address) >= 0)
return;
/* Can handle a stale RO->RW TLB */
if (spurious_fault(address, error_code))
return;
/* kprobes don't want to hook the spurious faults. */
if (notify_page_fault(regs))
return;
/*
* Don't take the mm semaphore here. If we fixup a prefetch
* fault we could otherwise deadlock.
*/
goto bad_area_nosemaphore;
}This includes vmalloc_fault. Note that I already looked into vmalloc_fault to ensure it could handle NMIs on x86_64 as well. See commit bdd5ea31e79fed76eb57d0cd797355267f4f4a8c. It seems I missed the cr2 register issue though. I assumed NMI handler to save it somehow, which ends up not being the case.
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 8 lines · Mathieu Desnoyers
I am not asking for the pf handler to handle every possible kind of fault recursively. Just to keep the in-kernel page fault related code for vmalloc (and possibly for prefetch ?) paths NMI-reentrant :
void
do_page_fault(struct pt_regs *regs, unsigned long error_code)address = read_cr2();
Why would this be needed? We read the cr2 as the first thing in do_page_fault(). It can be destroyed and re-faulted at will after that point, it wont matter a bit - we have already read it.
The only window to be careful about wrt. cr2 is the small window starting at <page_fault>, leading into <do_page_fault>:
ffffffff8154085f <do_page_fault>:
ffffffff8154085f: 55 push %rbp ffffffff81540860: 48 89 e5 mov %rsp,%rbp ffffffff81540863: 41 57 push %r15 ffffffff81540865: 41 56 push %r14 ffffffff81540867: 49 89 f6 mov %rsi,%r14 ffffffff8154086a: 41 55 push %r13 ffffffff8154086c: 49 89 fd mov %rdi,%r13 ffffffff8154086f: 41 54 push %r12 ffffffff81540871: 53 push %rbx ffffffff81540872: 48 83 ec 18 sub $0x18,%rsp ffffffff81540876: 65 4c 8b 3c 25 00 b0 mov %gs:0xb000,%r15 ffffffff8154087d: 00 00 ffffffff8154087f: 49 8b 87 48 02 00 00 mov 0x248(%r15),%rax ffffffff81540886: 48 89 45 d0 mov %rax,-0x30(%rbp) ffffffff8154088a: 48 83 c0 60 add $0x60,%rax ffffffff8154088e: 48 89 45 c8 mov %rax,-0x38(%rbp)
ffffffff81540892: 0f 18 08 prefetcht0 (%rax)ffffffff81540895: 41 0f 20 d4 mov %cr2,%r12
Look how early we read out cr2 - after trapping we read it after about 40 straight instructions, with no other function call inbetween. Only an NMI (or an MCE and similar deep-atomic contexts) can get in that window.
( Btw., a sidenote: the prefetcht0 right before the cr2 read is a
real bug. Prefetches can sometimes generate false faults and thus destroy the value cr2. I'll send a patch for that soon. )
Ingo
Quoted 8 lines · * Ingo Molnar ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
I am not asking for the pf handler to handle every possible kind of fault recursively. Just to keep the in-kernel page fault related code for vmalloc (and possibly for prefetch ?) paths NMI-reentrant :
Quoted 8 lines
Quoted 3 lines
void
do_page_fault(struct pt_regs *regs, unsigned long error_code)address = read_cr2();Why would this be needed? We read the cr2 as the first thing in
do_page_fault(). It can be destroyed and re-faulted at will after that point, it wont matter a bit - we have already read it.
With respect to cr2, yes, this is the only window we care about. However, the rest of vmalloc_fault() must be audited for other non nmi-suitable data structure use (e.g. "current"), which I did in the past.
My intent was just to respond to Peter's concerns by showing that the part of page fault handler which needs to be NMI-reentrant is really not that big.
Mathieu
Quoted 38 lines
The only window to be careful about wrt. cr2 is the small window starting at <
page_fault>, leading into <do_page_fault>:
ffffffff8154085f<do_page_fault>:ffffffff8154085f: 55 push %rbp ffffffff81540860: 48 89 e5 mov %rsp,%rbp ffffffff81540863: 41 57 push %r15 ffffffff81540865: 41 56 push %r14 ffffffff81540867: 49 89 f6 mov %rsi,%r14 ffffffff8154086a: 41 55 push %r13 ffffffff8154086c: 49 89 fd mov %rdi,%r13 ffffffff8154086f: 41 54 push %r12 ffffffff81540871: 53 push %rbx ffffffff81540872: 48 83 ec 18 sub $0x18,%rsp ffffffff81540876: 65 4c 8b 3c 25 00 b0 mov %gs:0xb000,%r15 ffffffff8154087d: 00 00 ffffffff8154087f: 49 8b 87 48 02 00 00 mov 0x248(%r15),%rax ffffffff81540886: 48 89 45 d0 mov %rax,-0x30(%rbp) ffffffff8154088a: 48 83 c0 60 add $0x60,%rax ffffffff8154088e: 48 89 45 c8 mov %rax,-0x38(%rbp)
ffffffff81540892: 0f 18 08 prefetcht0 (%rax)ffffffff81540895: 41 0f 20 d4 mov %cr2,%r12Look how early we read out cr2 - after trapping we read it after about 40 straight instructions, with no other function call inbetween. Only an NMI (or an MCE and similar deep-atomic contexts) can get in that window.
( Btw., a sidenote: the prefetcht0 right before the cr2 read is a
real bug. Prefetches can sometimes generate false faults and thus destroy the value cr2. I'll send a patch for that soon. )Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 10 lines · Mathieu Desnoyers
With respect to cr2, yes, this is the only window we care about. However, the rest of
vmalloc_fault()must be audited for other non nmi-suitable data structure use (e.g. "current"), which I did in the past.My intent was just to respond to Peter's concerns by showing that the part of page fault handler which needs to be NMI-reentrant is really not that big.
Even if that is true now, you want it to be true for all future time, and all to support an out-of-tree piece of code. All of this is virtually impossible to test for without said out-of-tree piece of code, so I will object to it anyway.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 12 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 10 lines · Mathieu Desnoyers
With respect to cr2, yes, this is the only window we care about. However, the rest of
vmalloc_fault()must be audited for other non nmi-suitable data structure use (e.g. "current"), which I did in the past.My intent was just to respond to Peter's concerns by showing that the part of page fault handler which needs to be NMI-reentrant is really not that big.
Quoted 5 lines
Even if that is true now, you want it to be true for all future time, and all to support an out-of-tree piece of code. All of this is virtually impossible to test for without said out-of-tree piece of code, so I will object to it anyway.
I think we are confusing two very distinct topics here :
LTTng is currently out-of-tree. Yes. It does not have to stay like this in the future. Or it can be a different tracer, like ftrace for instance.
LTTng can be built as modules. This is very likely to stay like this even if LTTng (or parts of) are merged. Or as a different tracer is merged. The reason why building a tracer as a module is convenient for users has been expressed in a previous mail.
So now you argue that it should not be made easy to implement tracers/profilers so they can be built as kernel modules because the LTTng tracer is out-of-tree. I'm sorry, but I really don't follow your line of reasoning.
So let's say we merge tracer or profiler code into the mainline kernel and permit that code to be built as module, hence enable testing within the mainline tree, would you be fine with this?
Mathieu
Quoted 6 lines
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 28 lines · Mathieu Desnoyers
Quoted 6 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
With respect to cr2, yes, this is the only window we care about. However, the rest of
vmalloc_fault()must be audited for other non nmi-suitable data structure use (e.g. "current"), which I did in the past.Quoted 9 lines
Quoted 4 lines
My intent was just to respond to Peter's concerns by showing that the part of page fault handler which needs to be NMI-reentrant is really not that big.
Even if that is true now, you want it to be true for all future time, and all to support an out-of-tree piece of code. All of this is virtually impossible to test for without said out-of-tree piece of code, so I will object to it anyway.
I think we are confusing two very distinct topics here :
LTTng is currently out-of-tree. Yes. It does not have to stay like this in the future. Or it can be a different tracer, like ftrace for instance.
LTTng can be built as modules. This is very likely to stay like this even if LTTng (or parts of) are merged. Or as a different tracer is merged. The reason why building a tracer as a module is convenient for users has been expressed in a previous mail.
Quoted 9 lines
So now you argue that it should not be made easy to implement tracers/profilers so they can be built as kernel modules because the LTTng tracer is out-of-tree. I'm sorry, but I really don't follow your line of reasoning.
So let's say we merge tracer or profiler code into the mainline kernel and permit that code to be built as module, hence enable testing within the mainline tree, would you be fine with this?
I'm saying that imposing constraints on kernel code has cost. The cost may not be immediately evident, but it constraints the kernel development going forward, potentially for all times. It is particularly obnoxious with out-of-tree users, because it is impossible to fix up those users to deal with a new interface, or even know what their constraints really are.
This is part of the fundamental problem that Xen causes, for example.
-hpa
Quoted 39 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 28 lines · Mathieu Desnoyers
Quoted 6 lines · * H. Peter Anvin ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
With respect to cr2, yes, this is the only window we care about. However, the rest of
vmalloc_fault()must be audited for other non nmi-suitable data structure use (e.g. "current"), which I did in the past.Quoted 9 lines
Quoted 4 lines
My intent was just to respond to Peter's concerns by showing that the part of page fault handler which needs to be NMI-reentrant is really not that big.
Even if that is true now, you want it to be true for all future time, and all to support an out-of-tree piece of code. All of this is virtually impossible to test for without said out-of-tree piece of code, so I will object to it anyway.
I think we are confusing two very distinct topics here :
LTTng is currently out-of-tree. Yes. It does not have to stay like this in the future. Or it can be a different tracer, like ftrace for instance.
LTTng can be built as modules. This is very likely to stay like this even if LTTng (or parts of) are merged. Or as a different tracer is merged. The reason why building a tracer as a module is convenient for users has been expressed in a previous mail.
Quoted 9 lines
So now you argue that it should not be made easy to implement tracers/profilers so they can be built as kernel modules because the LTTng tracer is out-of-tree. I'm sorry, but I really don't follow your line of reasoning.
So let's say we merge tracer or profiler code into the mainline kernel and permit that code to be built as module, hence enable testing within the mainline tree, would you be fine with this?
Quoted 9 lines
I'm saying that imposing constraints on kernel code has cost. The cost may not be immediately evident, but it constraints the kernel development going forward, potentially for all times. It is particularly obnoxious with out-of-tree users, because it is impossible to fix up those users to deal with a new interface, or even know what their constraints really are.
This is part of the fundamental problem that Xen causes, for example.
Agreed. And I agree that mainlining such users is a big part of the answer, because then it makes the whole community aware of their (ab)uses. However, wrt the specific case discussed here, I prefer by far adding a reentrancy constraint on a very well defined path of the trap handler if it permits to simplify a bunch of in-kernel users. This added encapsulation of architecture corner-cases will eventually make overall kernel development _simpler_, not harder.
Now if such encapsulation has an unbearable runtime cost, fine, we're big boys and we can tweak the kernel code to deal on a case-by-case basis with these corner-cases. However this kind of approach is usually more error-prone.
So, in summary :
- near-zero measurable runtime cost.
- NMI-reentrancy constraint on a very small and well-defined trap handler code path.
- simplifies life of tracer and profilers. (meaning : makes a lot of _other_ kernel code much easier to write and understand)
- removes ad-hoc corner cases management from those users.
- provides early error detection because the nmi-reentrant code path is shared by all users.
So I'll use your own argument : making this trap handler code path nmi-reentrant will simplify an already existing bunch of in-kernel users (oprofile, perf counter tool, ftrace..). Moving the burden from subsystems spread across the kernel tree to a single, well defined spot looks like a constraint that will _diminish_ overall kernel development cost.
Mathieu
Quoted 1 line
-hpa
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
H. Peter Anvin
reviewerQuoted 12 lines · Mathieu Desnoyers
So, in summary :
- near-zero measurable runtime cost.
- NMI-reentrancy constraint on a very small and well-defined trap handler code path.
- simplifies life of tracer and profilers. (meaning : makes a lot of _other_ kernel code much easier to write and understand)
- removes ad-hoc corner cases management from those users.
- provides early error detection because the nmi-reentrant code path is shared by all users.
Quoted 7 lines
So I'll use your own argument : making this trap handler code path nmi-reentrant will simplify an already existing bunch of in-kernel users (oprofile, perf counter tool, ftrace..). Moving the burden from subsystems spread across the kernel tree to a single, well defined spot looks like a constraint that will _diminish_ overall kernel development cost.
No, this is utter bullshit.
YOU ARE ADDING A CONSTRAINT TO ONE OF THE HOTTEST PATHS IN THE KERNEL.
Constraining future optimizations.
To support tools.
That is what I'm objecting to.
-hpa
[tip:x86/urgent] x86: mm: Read cr2 before prefetching the mmap_lock
In reply to Mathieu DesnoyersCommit-ID: 5dfaf90f8052327c92fbe3c470a2e6634be296c0 Gitweb: http://git.kernel.org/tip/5dfaf90f8052327c92fbe3c470a2e6634be296c0 Author: Ingo Molnar <[email protected]> AuthorDate: Tue, 16 Jun 2009 10:23:32 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 16 Jun 2009 10:23:32 +0200
x86: mm: Read cr2 before prefetching the mmap_lock
Prefetch instructions can generate spurious faults on certain models of older CPUs. The faults themselves cannot be stopped and they can occur pretty much anywhere - so the way we solve them is that we detect certain patterns and ignore the fault.
There is one small path of code where we must not take faults
though: the #PF handler execution leading up to the reading of the CR2 (the faulting address). If we take a fault there then we destroy the CR2 value (with that of the prefetching instruction's) and possibly mishandle user-space or kernel-space pagefaults.
It turns out that in current upstream we do exactly that:
prefetchw(&mm->mmap_sem); /* Get the faulting address: */ address = read_cr2();
This is not good.
So turn around the order: first read the cr2 then prefetch the lock address. Reading cr2 is plenty fast (2 cycles) so delaying the prefetch by this amount shouldnt be a big issue performance-wise.
[ And this might explain a mystery fault.c warning that sometimes
occurs on one an old AMD/Semptron based test-system i have - which does have such prefetch problems. ]
Cc: Mathieu Desnoyers <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Nick Piggin <[email protected]>
Cc: Pekka Enberg <[email protected]>
Cc: Vegard Nossum <[email protected]>
Cc: Jeremy Fitzhardinge <[email protected]>
Cc: Hugh Dickins <[email protected]>
LKML-Reference: <20090616030522.GA22162@Krystal>
--- arch/x86/mm/fault.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/fault.c +2 -2
@@ -951,11 +951,11 @@ do_page_fault(struct pt_regs *regs, unsigned long error_code) tsk = current; mm = tsk->mm; -prefetchw (&mm->mmap_sem); -/* Get the faulting address: */ address =read_cr2 (); +prefetchw (&mm->mmap_sem); +if (unlikely (kmmio_fault (regs, address)))return ;
Re: [tip:x86/urgent] x86: mm: Read cr2 before prefetching the mmap_lock
Quoted 9 lines · tip-bot for Ingo Molnar
It turns out that in current upstream we do exactly that:
prefetchw(&mm->mmap_sem); /* Get the faulting address: */ address = read_cr2();This is not good.
Ack.
Linus
Quoted 5 lines · Peter Zijlstra
Ah, well, I have no idea who expensive cr2 is, if its like a regular register then it should be fine. If however its tons more expensive then we should really avoid it.
IIRC it's pretty cheap, I measured it once when tuning the kvm context switch code. I don't recall the exact numbers.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
H. Peter Anvin
reviewerQuoted 8 lines · Avi Kivity
Quoted 3 lines · Peter Zijlstra
Ah, well, I have no idea who expensive cr2 is, if its like a regular register then it should be fine. If however its tons more expensive then we should really avoid it.
IIRC it's pretty cheap, I measured it once when tuning the kvm context switch code. I don't recall the exact numbers.
Sounds like a win to me.
-hpa
Quoted 5 lines · Mathieu Desnoyers
Quoted 4 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 2 lines · Mathieu Desnoyers
we should not care that much about the performance hit of saving/restoring the cr2 register at each nmi entry/exit.
Quoted 8 lines
Quoted 2 lines
But we do, perf counters very much cares about nmi performance.
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
I routinely run 'perf' with half a million NMIs per second or more. ( Why wait 10 seconds for a profile you can get in 1 second? ;-)
Granted that is over multiple CPUs - but still performance does matter here too.
Reading cr2 is certainly fast. Writing it - dunno.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 6 lines
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
Quoted 8 lines
I routinely run 'perf' with half a million NMIs per second or more. ( Why wait 10 seconds for a profile you can get in 1 second? ;-)
Granted that is over multiple CPUs - but still performance does matter here too.
Reading cr2 is certainly fast. Writing it - dunno.
But one thing is sure: it is certainly going to be faster than the INVLPG(s!) we have to do with the GUP solution.
Ingo
Quoted 13 lines · Ingo Molnar
Quoted 11 lines · Ingo Molnar
Quoted 6 lines
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
I routinely run 'perf' with half a million NMIs per second or more. ( Why wait 10 seconds for a profile you can get in 1 second? ;-)
Quoted 7 lines
Quoted 4 lines
Granted that is over multiple CPUs - but still performance does matter here too.
Reading cr2 is certainly fast. Writing it - dunno.
But one thing is sure: it is certainly going to be faster than the INVLPG(s!) we have to do with the GUP solution.
Sure, but we only pay that price when we do the callchain bit, not on every NMI.
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines
But one thing is sure: it is certainly going to be faster than the INVLPG(s!) we have to do with the GUP solution.
Sure, but we only pay that price when we do the callchain bit, not on every NMI.
You can read cr2 on nmi entry and nmi exit, and only write it if it has changed (if it turns out that writing cr2 is expensive; ISTR that it isn't).
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 2 lines · Avi Kivity
You can read cr2 on nmi entry and nmi exit, and only write it if it has changed (if it turns out that writing cr2 is expensive; ISTR that it isn't).
Writing the real cr2 is expensive, but it may be cheap to write the virtual cr2 in a vmx context. Or something.
J
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Ingo Molnar
Quoted 14 lines · Ingo Molnar
Quoted 6 lines
To a point where it cannot afford a simple register save/restore ?
There is "caring" and "_caring_". I am tempted to ask what NMI handler execution frequency you have in mind here to figure out if we are not trying to optimize sub-nanoseconds per minutes. ;)
I routinely run 'perf' with half a million NMIs per second or more. ( Why wait 10 seconds for a profile you can get in 1 second? ;-)
Granted that is over multiple CPUs - but still performance does matter here too.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Reading cr2 is certainly fast. Writing it - dunno.
But one thing is sure: it is certainly going to be faster than the INVLPG(s!) we have to do with the GUP solution.
Sure, but we only pay that price when we do the callchain bit, not on every NMI.
same goes for the CR2 save/restore trick - it only has to be done around the code where we expect to generate a #PF. I.e. in the call-graph bits.
Ingo