kcsan: Avoid scheduler recursion by using non-instrumented preempt_{disable,enable}()
Marco Elver
maintainerPATCH v2 1/1 kcsan: Avoid scheduler recursion by using non-instrumented preempt_{disable,enable}()
When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:
check_access() [via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add() [in kernel/sched/core.c]
check_access() [via instrumentation]
Avoid this by rewriting kcsan_prandom_u32_max() to only use safe versions of preempt_disable() and preempt_enable() that do not call into scheduler code.
Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed from kernel/sched/Makefile to permit testing scheduler code with KCSAN if desired.
---
v2:
* Update comment to also point out preempt_enable().
--- kernel/kcsan/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)
kernel/kcsan/core.c +12 -3
@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx **/ static u32 kcsan_prandom_u32_max (u32 ep_ro) { -struct rnd_state *state = &get_cpu_var (kcsan_rand_state); -const u32 res =prandom_u32_state (state); +struct rnd_state *state; +u32 res; + +/* +* Avoid recursion with scheduler by using non-tracing versions of +* preempt_disable() and preempt_enable() that do not call into +* scheduler code. +*/ +preempt_disable_notrace (); + state =raw_cpu_ptr (&kcsan_rand_state); + res =prandom_u32_state (state); +preempt_enable_no_resched_notrace (); -put_cpu_var (kcsan_rand_state);return (u32)(((u64 ) res * ep_ro) >>32 ); }
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
Peter Zijlstra
maintainerQuoted 43 lines · Marco Elver
When enabling KCSAN for kernel/sched (remove
KCSAN_SANITIZE:= n fromkernel/sched/Makefile), withCONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:
check_access()[via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add()[inkernel/sched/core.c]
check_access()[via instrumentation]Avoid this by rewriting
kcsan_prandom_u32_max()to only use safe versions ofpreempt_disable()andpreempt_enable()that do not call into scheduler code.Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when
KCSAN_SANITIZE:= n is removed fromkernel/sched/Makefileto permit testing scheduler code with KCSAN if desired.Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")Signed-off-by Marco Elver <[email protected]>---
v2:
* Update comment to also point outpreempt_enable().--- kernel/kcsan/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)kernel/kcsan/core.c +2
-2@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx **/ static u32 kcsan_prandom_u32_max (u32 ep_ro) { -struct rnd_state *state = &get_cpu_var (kcsan_rand_state); -const u32 res =prandom_u32_state (state); +struct rnd_state *state; +u32 res;
kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298
Quoted 10 lines
kernel/kcsan/core.c +10
-0+ +/* +* Avoid recursion with scheduler by using non-tracing versions of +* preempt_disable() and preempt_enable() that do not call into +* scheduler code. +*/ +preempt_disable_notrace (); + state =raw_cpu_ptr (&kcsan_rand_state); + res =prandom_u32_state (state); +preempt_enable_no_resched_notrace ();
This is a preemption bug. Does preempt_enable_notrace() not work?
Quoted 8 lines
kernel/kcsan/core.c +0
-1-put_cpu_var (kcsan_rand_state);return (u32)(((u64 ) res * ep_ro) >>32 ); }Signature · 1 line
2.29.2.454.gaff20da3a2-goog
Marco Elver
maintainerQuoted 48 lines · Peter Zijlstra
Quoted 47 lines · Marco Elver
When enabling KCSAN for kernel/sched (remove
KCSAN_SANITIZE:= n fromkernel/sched/Makefile), withCONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:
check_access()[via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add()[inkernel/sched/core.c]
check_access()[via instrumentation]Avoid this by rewriting
kcsan_prandom_u32_max()to only use safe versions ofpreempt_disable()andpreempt_enable()that do not call into scheduler code.Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when
KCSAN_SANITIZE:= n is removed fromkernel/sched/Makefileto permit testing scheduler code with KCSAN if desired.Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")Signed-off-by Marco Elver <[email protected]>---
v2:
* Update comment to also point outpreempt_enable().--- kernel/kcsan/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)kernel/kcsan/core.c +6
-2@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx **/ static u32 kcsan_prandom_u32_max (u32 ep_ro) { -struct rnd_state *state = &get_cpu_var (kcsan_rand_state); -const u32 res =prandom_u32_state (state); +struct rnd_state *state; +u32 res; + +/* +* Avoid recursion with scheduler by using non-tracing versions of +* preempt_disable() and preempt_enable() that do not call into
Quoted 8 lines
kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298
Quoted 6 lines
kernel/kcsan/core.c +6
-0+* scheduler code. +*/ +preempt_disable_notrace (); + state =raw_cpu_ptr (&kcsan_rand_state); + res =prandom_u32_state (state); +preempt_enable_no_resched_notrace ();This is a preemption bug. Does
preempt_enable_notrace()not work?
No it didn't, because we end up calling preempt_schedule_notrace(), which again might end in recursion.
Normally we could surround this by kcsan_disable_current/kcsan_enable_current(), but that doesn't work because we have this sequence:
reset_kcsan_skip();
if (!kcsan_is_enabled())
...to avoid underflowing the skip counter if KCSAN is disabled. That could be solved by writing to the skip-counter twice: once with a non-random value, and if KCSAN is enabled with a random value. Would that be better?
And I'd like to avoid adding __no_kcsan to scheduler functions.
Any recommendation?
Thanks,
-- Marco
Quoted 9 lines
Quoted 9 lines
kernel/kcsan/core.c +0
-2-put_cpu_var (kcsan_rand_state);return (u32)(((u64 ) res * ep_ro) >>32 ); } --2.29.2.454.gaff20da3a2-goog
Marco Elver
maintainerQuoted 65 lines · Marco Elver
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Marco Elver
When enabling KCSAN for kernel/sched (remove
KCSAN_SANITIZE:= n fromkernel/sched/Makefile), withCONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:
check_access()[via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add()[inkernel/sched/core.c]
check_access()[via instrumentation]Avoid this by rewriting
kcsan_prandom_u32_max()to only use safe versions ofpreempt_disable()andpreempt_enable()that do not call into scheduler code.Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when
KCSAN_SANITIZE:= n is removed fromkernel/sched/Makefileto permit testing scheduler code with KCSAN if desired.Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")Signed-off-by Marco Elver <[email protected]>---
v2:
* Update comment to also point outpreempt_enable().--- kernel/kcsan/core.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-)kernel/kcsan/core.c +6
-2@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx **/ static u32 kcsan_prandom_u32_max (u32 ep_ro) { -struct rnd_state *state = &get_cpu_var (kcsan_rand_state); -const u32 res =prandom_u32_state (state); +struct rnd_state *state; +u32 res; + +/* +* Avoid recursion with scheduler by using non-tracing versions of +* preempt_disable() and preempt_enable() that do not call into Quoted 8 lines
kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298
Quoted 6 lines
kernel/kcsan/core.c +6
-0+* scheduler code. +*/ +preempt_disable_notrace (); + state =raw_cpu_ptr (&kcsan_rand_state); + res =prandom_u32_state (state); +preempt_enable_no_resched_notrace ();This is a preemption bug. Does
preempt_enable_notrace()not work?No it didn't, because we end up calling
preempt_schedule_notrace(), which again might end in recursion.Normally we could surround this by kcsan_disable_current/kcsan_enable_current(), but that doesn't work because we have this sequence:
Quoted 8 lines
reset_kcsan_skip(); if (!kcsan_is_enabled()) ...to avoid underflowing the skip counter if KCSAN is disabled. That could be solved by writing to the skip-counter twice: once with a non-random value, and if KCSAN is enabled with a random value. Would that be better?
See below for concrete alternative that works.
Quoted 3 lines
And I'd like to avoid adding
__no_kcsanto scheduler functions.Any recommendation?
Let me know what you prefer.
Thanks,
-- Marco
------ >8 ------
kernel/kcsan/core.c +16 -18
@@ -266,8 +266,8 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *return false;/* -* NOTE: If we get here, kcsan_skip must always be reset in slow path -* via reset_kcsan_skip() to avoid underflow. +* Note: If we get here, kcsan_skip must always be reset in slow path to +* avoid underflow. */ /* this operation should be watched */
@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)u32 res;/* -* Avoid recursion with scheduler by using non-tracing versions of -* preempt_disable() and preempt_enable() that do not call into -* scheduler code. +* Avoid recursion with scheduler by disabling KCSAN because +* preempt_enable_notrace() will still call into scheduler code. */ +kcsan_disable_current ();preempt_disable_notrace (); state =raw_cpu_ptr (&kcsan_rand_state); res =prandom_u32_state (state); -preempt_enable_no_resched_notrace (); +preempt_enable_notrace (); +kcsan_enable_current_nowarn ();return (u32)(((u64 ) res * ep_ro) >>32 ); } -static inline void reset_kcsan_skip (void ) -{ -long skip_count = kcsan_skip_watch - - (IS_ENABLED (CONFIG_KCSAN_SKIP_WATCH_RANDOMIZE ) ? -kcsan_prandom_u32_max (kcsan_skip_watch) : -0 ); -this_cpu_write (kcsan_skip, skip_count); -} - static __always_inline bool kcsan_is_enabled(void) {return READ_ONCE (kcsan_enabled) &&get_ctx ()->disable_count ==0 ;
@@ -430,10 +422,16 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type)* Always reset kcsan_skip counter in slow-path to avoid underflow; see * should_watch(). */ -reset_kcsan_skip (); - -if (!kcsan_is_enabled ()) +if (likely (kcsan_is_enabled ())) { +long skip_count = kcsan_skip_watch - + (IS_ENABLED (CONFIG_KCSAN_SKIP_WATCH_RANDOMIZE ) ? +kcsan_prandom_u32_max (kcsan_skip_watch) : +0 ); +this_cpu_write (kcsan_skip, skip_count); + }else { +this_cpu_write (kcsan_skip, kcsan_skip_watch); goto out; + }/* * Special atomic rules: unlikely to be true, so we check them here in
kernel/sched/Makefile +0 -6
@@ -7,12 +7,6 @@ endif # that is not a function of syscall inputs. E.g. involuntary context switches. KCOV_INSTRUMENT := n -# There are numerous data races here, however, most of them are due to plain accesses. -# This would make it even harder for syzbot to find reproducers, because these -# bugs trigger without specific input. Disable by default, but should re-enable -# eventually. -KCSAN_SANITIZE := n - ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is # needed for x86 only. Why this used to be enabled for all architectures is beyond
Peter Zijlstra
maintainerQuoted 9 lines · Marco Elver
Let me know what you prefer.
kernel/kcsan/core.c +2
-0@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)u32 res;/* +* Avoid recursion with scheduler by disabling KCSAN because +* preempt_enable_notrace() will still call into scheduler code.
kernel/kcsan/core.c static u32 kcsan_prandom_u32_max(u32 ep_ro) · line 302
Quoted 10 lines
kernel/kcsan/core.c +3
-0*/ +kcsan_disable_current ();preempt_disable_notrace (); state =raw_cpu_ptr (&kcsan_rand_state); res =prandom_u32_state (state); +preempt_enable_notrace (); +kcsan_enable_current_nowarn ();return (u32)(((u64 ) res * ep_ro) >>32 ); }
This is much preferred over the other. The thing with _no_resched is that you can miss a preemption for an unbounded amount of time, which is bad.
The _only_ valid use of _no_resched is when there's a call to schedule() right after it.
Marco Elver
maintainerQuoted 18 lines · Peter Zijlstra
Quoted 17 lines · Marco Elver
Let me know what you prefer.
kernel/kcsan/core.c +5
-0@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)u32 res;/* +* Avoid recursion with scheduler by disabling KCSAN because +* preempt_enable_notrace() will still call into scheduler code. */ +kcsan_disable_current ();preempt_disable_notrace (); state =raw_cpu_ptr (&kcsan_rand_state); res =prandom_u32_state (state); +preempt_enable_notrace (); +kcsan_enable_current_nowarn ();
Quoted 5 lines
kernel/kcsan/core.c static u32 kcsan_prandom_u32_max(u32 ep_ro) · line 302
Quoted 2 lines
kernel/kcsan/core.c +0
-0return (u32)(((u64 ) res * ep_ro) >>32 ); }This is much preferred over the other. The thing with
_no_reschedis that you can miss a preemption for an unbounded amount of time, which is bad.
Ah, I think this is rubbish, too. Because it might fix one problem, but now I'm left with the problem that kcsan_prandom_u32_max() is called for udelay() later and at that point we lose skip_count randomness entirely.
I think relying on lib/random32.c already caused too many headaches, so I'm tempted to just get rid of that dependency entirely. And instead do the simplest possible thing, which might just be calling get_cycles() (all we need is to introduce some non-determinism).
Quoted 2 lines
The _only_ valid use of
_no_reschedis when there's a call toschedule()right after it.