preempt: Fix tracing crash
Subsystems (3)
cover preempt: Fix tracing crash
This is a proposition to fix the function graph tracer crashes reported by Fengguang Wu's 0-day testing.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
sched/core
HEAD: 723d604e2c185b94c6890c26dab9c72e4243711b
Thanks,
Frederic
---
Frederic Weisbecker (3):
sched: Make preempt_schedule_context() function-tracing safe
preempt: Use preempt_schedule_context() as the official tracing preemption point
preempt: Reorganize a bit the notrace declensions
arch/x86/include/asm/preempt.h | 8 +++----- arch/x86/kernel/i386_ksyms_32.c | 4 +--- arch/x86/kernel/x8664_ksyms_64.c | 4 +--- arch/x86/lib/thunk_32.S | 4 +--- arch/x86/lib/thunk_64.S | 4 +--- include/asm-generic/preempt.h | 7 ++----- include/linux/preempt.h | 36 +++++++++++++++--------------------- kernel/sched/core.c | 19 ++++++++++++------- 8 files changed, 36 insertions(+), 50 deletions(-)
PATCH 1/3 sched: Make preempt_schedule_context() function-tracing safe
Since function tracing disables preemption, it needs a safe preemption point to use when preemption is re-enabled without worrying about tracing recursion. Ie: to avoid tracing recursion, that preemption point can't be traced (use of notrace qualifier) and it can't call any traceable function before that preemption point disables preemption itself, which disarms the recursion.
preempt_schedule() was fine until commit b30f0e3ffedfa52b1d67a302ae5860c49998e5e2
("sched/preempt: Optimize preemption operations on __schedule() callers") because PREEMPT_ACTIVE (which has the property to disable preemption and this disarm tracing preemption recursion) was set before calling any further function.
But that commit introduced the use of preempt_count_add/sub() functions to set PREEMPT_ACTIVE and because these functions are called before preemption gets a chance to be disabled, we have a tracing recursion.
preempt_schedule_context() is one of the possible preemption functions used by tracing. Its special purpose is to avoid tracing recursion against context tracking. Lets enhance this function to become more generally tracing safe by disabling preemption with raw accessors, such that no function is called before preemption gets disabled and disarm the tracing recursion.
This function is going to become the specific tracing-safe preemption point in further commit.
--- kernel/sched/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +9 -2
@@ -2960,7 +2960,13 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)return ; do { - preempt_active_enter(); + /* + * Use raw __prempt_count() ops that don't call function. + * We can't call functions before disabling preemption which + * disarm preemption tracing recursions. + */ + __preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); + barrier(); /* * Needs preempt disabled in case user_exit() is traced * and the tracer calls preempt_enable_notrace() causing
@@ -2970,7 +2976,8 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)__schedule ();exception_exit (prev_ctx); -preempt_active_exit (); +barrier (); +__preempt_count_sub (PREEMPT_ACTIVE +PREEMPT_DISABLE_OFFSET ); }while (need_resched ()); }EXPORT_SYMBOL_GPL (preempt_schedule_context);
Signature · 1 line
2.1.4
tip-bot for Frederic Weisbecker
merge bot[tip:sched/core] sched: Make preempt_schedule_context() function-tracing safe
sched: Make preempt_schedule_context() function-tracing safe
Commit-ID: be690035df893385ceaac2323b29be1fb7f2a67f Gitweb: http://git.kernel.org/tip/be690035df893385ceaac2323b29be1fb7f2a67f Author: Frederic Weisbecker <[email protected]> AuthorDate: Thu, 4 Jun 2015 17:39:07 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 7 Jun 2015 15:57:41 +0200
sched: Make preempt_schedule_context() function-tracing safe
Since function tracing disables preemption, it needs a safe preemption point to use when preemption is re-enabled without worrying about tracing recursion. Ie: to avoid tracing recursion, that preemption point can't be traced (use of notrace qualifier) and it can't call any traceable function before that preemption point disables preemption itself, which disarms the recursion.
preempt_schedule() was fine until commit:
b30f0e3ffedf ("sched/preempt: Optimize preemption operations on __schedule() callers")
because PREEMPT_ACTIVE (which has the property to disable preemption and this disarm tracing preemption recursion) was set before calling any further function.
But that commit introduced the use of preempt_count_add/sub() functions to set PREEMPT_ACTIVE and because these functions are called before preemption gets a chance to be disabled, we have a tracing recursion.
preempt_schedule_context() is one of the possible preemption functions used by tracing. Its special purpose is to avoid tracing recursion against context tracking. Lets enhance this function to become more generally tracing safe by disabling preemption with raw accessors, such that no function is called before preemption gets disabled and disarm the tracing recursion.
This function is going to become the specific tracing-safe preemption point in further commit.
--- kernel/sched/core.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/core.c +9 -2
@@ -2960,7 +2960,13 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)return ; do { - preempt_active_enter(); + /* + * Use raw __prempt_count() ops that don't call function. + * We can't call functions before disabling preemption which + * disarm preemption tracing recursions. + */ + __preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); + barrier(); /* * Needs preempt disabled in case user_exit() is traced * and the tracer calls preempt_enable_notrace() causing
@@ -2970,7 +2976,8 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void)__schedule ();exception_exit (prev_ctx); -preempt_active_exit (); +barrier (); +__preempt_count_sub (PREEMPT_ACTIVE +PREEMPT_DISABLE_OFFSET ); }while (need_resched ()); }EXPORT_SYMBOL_GPL (preempt_schedule_context);
PATCH 2/3 preempt: Use preempt_schedule_context() as the official tracing preemption point
In reply to Frederic Weisbeckerpreempt_schedule_context() is a tracing safe preemption point but it's only used when CONFIG_CONTEXT_TRACKING=y. Other configs have tracing recursion issues since b30f0e3ffedfa52b1d67a302ae5860c49998e5e2 ("sched/preempt: Optimize preemption operations on __schedule() callers") introduced function based preemp_coun_*() ops.
Lets make it available on all configs and give it a more appropriate name for its new position.
--- arch/x86/include/asm/preempt.h | 8 +++----- arch/x86/kernel/i386_ksyms_32.c | 4 +--- arch/x86/kernel/x8664_ksyms_64.c | 4 +--- arch/x86/lib/thunk_32.S | 4 +--- arch/x86/lib/thunk_64.S | 4 +--- include/asm-generic/preempt.h | 7 ++----- include/linux/preempt.h | 6 +----- kernel/sched/core.c | 8 +++----- 8 files changed, 13 insertions(+), 32 deletions(-)
arch/x86/include/asm/preempt.h +3 -5
@@ -99,11 +99,9 @@ static __always_inline bool should_resched(void) extern asmlinkage void ___preempt_schedule(void);# define __preempt_schedule () asm ("call ___preempt_schedule") extern asmlinkage void preempt_schedule(void); -# ifdef CONFIG_CONTEXT_TRACKING - extern asmlinkage void ___preempt_schedule_context(void); -# define __preempt_schedule_context () asm ("call ___preempt_schedule_context") - extern asmlinkage void preempt_schedule_context(void); -# endif + extern asmlinkage void ___preempt_schedule_notrace(void); +# define __preempt_schedule_notrace () asm ("call ___preempt_schedule_notrace") + extern asmlinkage void preempt_schedule_notrace(void);#endif #endif /* __ASM_PREEMPT_H */
arch/x86/kernel/i386_ksyms_32.c +1 -3
@@ -40,7 +40,5 @@ EXPORT_SYMBOL(empty_zero_page);#ifdef CONFIG_PREEMPT EXPORT_SYMBOL (___preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING -EXPORT_SYMBOL (___preempt_schedule_context); -#endif +EXPORT_SYMBOL (___preempt_schedule_notrace);#endif
arch/x86/kernel/x8664_ksyms_64.c +1 -3
@@ -75,7 +75,5 @@ EXPORT_SYMBOL(native_load_gs_index);#ifdef CONFIG_PREEMPT EXPORT_SYMBOL (___preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING -EXPORT_SYMBOL (___preempt_schedule_context); -#endif +EXPORT_SYMBOL (___preempt_schedule_notrace);#endif
arch/x86/lib/thunk_32.S +1 -3
@@ -38,8 +38,6 @@ #ifdef CONFIG_PREEMPT THUNK ___preempt_schedule, preempt_schedule -#ifdef CONFIG_CONTEXT_TRACKING - THUNK ___preempt_schedule_context, preempt_schedule_context -#endif + THUNK ___preempt_schedule_notrace, preempt_schedule_notrace #endif
arch/x86/lib/thunk_64.S +1 -3
@@ -49,9 +49,7 @@ #ifdef CONFIG_PREEMPT THUNK ___preempt_schedule, preempt_schedule -#ifdef CONFIG_CONTEXT_TRACKING - THUNK ___preempt_schedule_context, preempt_schedule_context -#endif + THUNK ___preempt_schedule_notrace, preempt_schedule_notrace #endif #if defined(CONFIG_TRACE_IRQFLAGS) \
include/asm-generic/preempt.h +2 -5
@@ -79,11 +79,8 @@ static __always_inline bool should_resched(void)#ifdef CONFIG_PREEMPT extern asmlinkage void preempt_schedule(void);#define __preempt_schedule () preempt_schedule() - -#ifdef CONFIG_CONTEXT_TRACKING -extern asmlinkage void preempt_schedule_context(void); -#define __preempt_schedule_context () preempt_schedule_context() -#endif +extern asmlinkage void preempt_schedule_notrace(void); +#define __preempt_schedule_notrace () preempt_schedule_notrace()#endif /* CONFIG_PREEMPT */ #endif /* __ASM_PREEMPT_H */
include/linux/preempt.h +1 -5
@@ -204,15 +204,11 @@ do { \#ifdef CONFIG_PREEMPT -#ifndef CONFIG_CONTEXT_TRACKING -#define __preempt_schedule_context () __preempt_schedule() -#endif -#define preempt_enable_notrace () \ do { \ barrier(); \ if (unlikely(__preempt_count_dec_and_test())) \ - __preempt_schedule_context(); \ + __preempt_schedule_notrace(); \ } while (0)#else #define preempt_enable_notrace () \
kernel/sched/core.c +3 -5
@@ -2937,9 +2937,8 @@ asmlinkage __visible void __sched notrace preempt_schedule(void)NOKPROBE_SYMBOL (preempt_schedule);EXPORT_SYMBOL (preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING /** -* preempt_schedule_context - preempt_schedule called by tracing +* preempt_schedule_notrace - preempt_schedule called by tracing * * The tracing infrastructure uses preempt_enable_notrace to prevent * recursion and tracing preempt enabling caused by the tracing
@@ -2952,7 +2951,7 @@ EXPORT_SYMBOL(preempt_schedule);* instead of preempt_schedule() to exit user context if needed before * calling the scheduler. */ -asmlinkage __visible void __sched notrace preempt_schedule_context(void) +asmlinkage __visible void __sched notrace preempt_schedule_notrace(void) {enum ctx_state prev_ctx;
@@ -2980,8 +2979,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void) __preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); }while (need_resched ()); } -EXPORT_SYMBOL_GPL (preempt_schedule_context); -#endif /* CONFIG_CONTEXT_TRACKING */ +EXPORT_SYMBOL_GPL (preempt_schedule_notrace);#endif /* CONFIG_PREEMPT */
Signature · 1 line
2.1.4
tip-bot for Frederic Weisbecker
merge bot[tip:sched/core] preempt: Use preempt_schedule_context() as the official tracing preemption point
preempt: Use preempt_schedule_context() as the official tracing preemption point
Commit-ID: 4eaca0a887eaee04fc7a3866d0f5b51b34030dfa Gitweb: http://git.kernel.org/tip/4eaca0a887eaee04fc7a3866d0f5b51b34030dfa Author: Frederic Weisbecker <[email protected]> AuthorDate: Thu, 4 Jun 2015 17:39:08 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 7 Jun 2015 15:57:42 +0200
preempt: Use preempt_schedule_context() as the official tracing preemption point
preempt_schedule_context() is a tracing safe preemption point but it's only used when CONFIG_CONTEXT_TRACKING=y. Other configs have tracing recursion issues since commit:
b30f0e3ffedf ("sched/preempt: Optimize preemption operations on __schedule() callers")
introduced function based preemp_count_*() ops.
Lets make it available on all configs and give it a more appropriate name for its new position.
--- arch/x86/include/asm/preempt.h | 8 +++----- arch/x86/kernel/i386_ksyms_32.c | 4 +--- arch/x86/kernel/x8664_ksyms_64.c | 4 +--- arch/x86/lib/thunk_32.S | 4 +--- arch/x86/lib/thunk_64.S | 4 +--- include/asm-generic/preempt.h | 7 ++----- include/linux/preempt.h | 6 +----- kernel/sched/core.c | 8 +++----- 8 files changed, 13 insertions(+), 32 deletions(-)
arch/x86/include/asm/preempt.h +3 -5
@@ -99,11 +99,9 @@ static __always_inline bool should_resched(void) extern asmlinkage void ___preempt_schedule(void);# define __preempt_schedule () asm ("call ___preempt_schedule") extern asmlinkage void preempt_schedule(void); -# ifdef CONFIG_CONTEXT_TRACKING - extern asmlinkage void ___preempt_schedule_context(void); -# define __preempt_schedule_context () asm ("call ___preempt_schedule_context") - extern asmlinkage void preempt_schedule_context(void); -# endif + extern asmlinkage void ___preempt_schedule_notrace(void); +# define __preempt_schedule_notrace () asm ("call ___preempt_schedule_notrace") + extern asmlinkage void preempt_schedule_notrace(void);#endif #endif /* __ASM_PREEMPT_H */
arch/x86/kernel/i386_ksyms_32.c +1 -3
@@ -40,7 +40,5 @@ EXPORT_SYMBOL(empty_zero_page);#ifdef CONFIG_PREEMPT EXPORT_SYMBOL (___preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING -EXPORT_SYMBOL (___preempt_schedule_context); -#endif +EXPORT_SYMBOL (___preempt_schedule_notrace);#endif
arch/x86/kernel/x8664_ksyms_64.c +1 -3
@@ -75,7 +75,5 @@ EXPORT_SYMBOL(native_load_gs_index);#ifdef CONFIG_PREEMPT EXPORT_SYMBOL (___preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING -EXPORT_SYMBOL (___preempt_schedule_context); -#endif +EXPORT_SYMBOL (___preempt_schedule_notrace);#endif
arch/x86/lib/thunk_32.S +1 -3
@@ -38,8 +38,6 @@ #ifdef CONFIG_PREEMPT THUNK ___preempt_schedule, preempt_schedule -#ifdef CONFIG_CONTEXT_TRACKING - THUNK ___preempt_schedule_context, preempt_schedule_context -#endif + THUNK ___preempt_schedule_notrace, preempt_schedule_notrace #endif
arch/x86/lib/thunk_64.S +1 -3
@@ -49,9 +49,7 @@ #ifdef CONFIG_PREEMPT THUNK ___preempt_schedule, preempt_schedule -#ifdef CONFIG_CONTEXT_TRACKING - THUNK ___preempt_schedule_context, preempt_schedule_context -#endif + THUNK ___preempt_schedule_notrace, preempt_schedule_notrace #endif #if defined(CONFIG_TRACE_IRQFLAGS) \
include/asm-generic/preempt.h +2 -5
@@ -79,11 +79,8 @@ static __always_inline bool should_resched(void)#ifdef CONFIG_PREEMPT extern asmlinkage void preempt_schedule(void);#define __preempt_schedule () preempt_schedule() - -#ifdef CONFIG_CONTEXT_TRACKING -extern asmlinkage void preempt_schedule_context(void); -#define __preempt_schedule_context () preempt_schedule_context() -#endif +extern asmlinkage void preempt_schedule_notrace(void); +#define __preempt_schedule_notrace () preempt_schedule_notrace()#endif /* CONFIG_PREEMPT */ #endif /* __ASM_PREEMPT_H */
include/linux/preempt.h +1 -5
@@ -204,15 +204,11 @@ do { \#ifdef CONFIG_PREEMPT -#ifndef CONFIG_CONTEXT_TRACKING -#define __preempt_schedule_context () __preempt_schedule() -#endif -#define preempt_enable_notrace () \ do { \ barrier(); \ if (unlikely(__preempt_count_dec_and_test())) \ - __preempt_schedule_context(); \ + __preempt_schedule_notrace(); \ } while (0)#else #define preempt_enable_notrace () \
kernel/sched/core.c +3 -5
@@ -2937,9 +2937,8 @@ asmlinkage __visible void __sched notrace preempt_schedule(void)NOKPROBE_SYMBOL (preempt_schedule);EXPORT_SYMBOL (preempt_schedule); -#ifdef CONFIG_CONTEXT_TRACKING /** -* preempt_schedule_context - preempt_schedule called by tracing +* preempt_schedule_notrace - preempt_schedule called by tracing * * The tracing infrastructure uses preempt_enable_notrace to prevent * recursion and tracing preempt enabling caused by the tracing
@@ -2952,7 +2951,7 @@ EXPORT_SYMBOL(preempt_schedule);* instead of preempt_schedule() to exit user context if needed before * calling the scheduler. */ -asmlinkage __visible void __sched notrace preempt_schedule_context(void) +asmlinkage __visible void __sched notrace preempt_schedule_notrace(void) {enum ctx_state prev_ctx;
@@ -2980,8 +2979,7 @@ asmlinkage __visible void __sched notrace preempt_schedule_context(void) __preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); }while (need_resched ()); } -EXPORT_SYMBOL_GPL (preempt_schedule_context); -#endif /* CONFIG_CONTEXT_TRACKING */ +EXPORT_SYMBOL_GPL (preempt_schedule_notrace);#endif /* CONFIG_PREEMPT */
PATCH 3/3 preempt: Reorganize a bit the notrace declensions
In reply to Frederic Weisbeckerpreempt.h has two seperate "#ifdef CONFIG_PREEMPT" sections: one to define preempt_enable() and another to define preempt_enable_notrace().
Lets gather both.
--- include/linux/preempt.h | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-)
include/linux/preempt.h +16 -18
@@ -175,20 +175,34 @@ do { \__preempt_schedule (); \ } while (0) +#define preempt_enable_notrace () \ +do { \ + barrier(); \ + if (unlikely(__preempt_count_dec_and_test())) \ + __preempt_schedule_notrace(); \ +} while (0) +#define preempt_check_resched () \ do { \ if (should_resched()) \ __preempt_schedule(); \ } while (0) -#else +#else /* !CONFIG_PREEMPT */ #define preempt_enable () \ do { \ barrier(); \ preempt_count_dec(); \ } while (0) + +#define preempt_enable_notrace () \ +do { \ + barrier(); \ + __preempt_count_dec(); \ +} while (0) +#define preempt_check_resched () do { } while (0) -#endif +#endif /* CONFIG_PREEMPT */ #define preempt_disable_notrace () \ do { \
@@ -202,22 +216,6 @@ do { \__preempt_count_dec (); \ } while (0) -#ifdef CONFIG_PREEMPT - -#define preempt_enable_notrace () \ -do { \ - barrier(); \ - if (unlikely(__preempt_count_dec_and_test())) \ - __preempt_schedule_notrace(); \ -} while (0) -#else -#define preempt_enable_notrace () \ -do { \ - barrier(); \ - __preempt_count_dec(); \ -} while (0) -#endif -#else /* !CONFIG_PREEMPT_COUNT */ /*
Signature · 1 line
2.1.4
tip-bot for Frederic Weisbecker
merge bot[tip:sched/core] preempt: Reorganize the notrace definitions a bit
preempt: Reorganize the notrace definitions a bit
Commit-ID: 9a92e3dc6ad02208a014d0d8404ebbd697e3d5ef Gitweb: http://git.kernel.org/tip/9a92e3dc6ad02208a014d0d8404ebbd697e3d5ef Author: Frederic Weisbecker <[email protected]> AuthorDate: Thu, 4 Jun 2015 17:39:09 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 7 Jun 2015 15:57:43 +0200
preempt: Reorganize the notrace definitions a bit
preempt.h has two seperate "#ifdef CONFIG_PREEMPT" sections: one to define preempt_enable() and another to define preempt_enable_notrace().
Lets gather both.
--- include/linux/preempt.h | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-)
include/linux/preempt.h +15 -17
@@ -175,48 +175,46 @@ do { \__preempt_schedule (); \ } while (0) +#define preempt_enable_notrace () \ +do { \ + barrier(); \ + if (unlikely(__preempt_count_dec_and_test())) \ + __preempt_schedule_notrace(); \ +} while (0) +#define preempt_check_resched () \ do { \ if (should_resched()) \ __preempt_schedule(); \ } while (0) -#else +#else /* !CONFIG_PREEMPT */ #define preempt_enable () \ do { \ barrier(); \ preempt_count_dec(); \ } while (0) -#define preempt_check_resched () do { } while (0) -#endif - -#define preempt_disable_notrace () \ -do { \ - __preempt_count_inc(); \ - barrier(); \ -} while (0) -#define preempt_enable_no_resched_notrace () \ +#define preempt_enable_notrace () \ do { \ barrier(); \ __preempt_count_dec(); \ } while (0) -#ifdef CONFIG_PREEMPT +#define preempt_check_resched () do { } while (0) +#endif /* CONFIG_PREEMPT */ -#define preempt_enable_notrace () \ +#define preempt_disable_notrace () \ do { \ + __preempt_count_inc(); \ barrier(); \ - if (unlikely(__preempt_count_dec_and_test())) \ - __preempt_schedule_notrace(); \ } while (0) -#else -#define preempt_enable_notrace () \ + +#define preempt_enable_no_resched_notrace () \ do { \ barrier(); \ __preempt_count_dec(); \ } while (0) -#endif #else /* !CONFIG_PREEMPT_COUNT */
Peter Zijlstra
maintainerQuoted 8 lines · Frederic Weisbecker
This is a proposition to fix the function graph tracer crashes reported by Fengguang Wu's 0-day testing.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
sched/coreHEAD:
723d604e2c185b94c6890c26dab9c72e4243711b
Quoted 9 lines
Thanks,
Frederic
---Frederic Weisbecker (3):
sched: Makepreempt_schedule_context()function-tracing safe
preempt: Usepreempt_schedule_context()as the official tracing preemption point
preempt: Reorganize a bit the notrace declensions
Thanks!