Skip to contentSkip to conversation

kcsan: Avoid scheduler recursion by using non-instrumented preempt_{disable,enable}()

PATCH In review Marco Elver 2 participants 6 messages Nov 23, 2020

Subsystems (2)
lore
Share & tools
lore

Marco Elver

author maintainer
#0 lore raw

PATCH v2 1/1 kcsan: Avoid scheduler recursion by using non-instrumented preempt_{disable,enable}()

When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:

check_access() [via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add() [in kernel/sched/core.c]
check_access() [via instrumentation]

Avoid this by rewriting kcsan_prandom_u32_max() to only use safe versions of preempt_disable() and preempt_enable() that do not call into scheduler code.

Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed from kernel/sched/Makefile to permit testing scheduler code with KCSAN if desired.

Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")

---
v2:
* Update comment to also point out preempt_enable().

---
 kernel/kcsan/core.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
kernel/kcsan/core.c +12 -3
index 3994a217bde7..10513f3e2349 100644
@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
  */
 static u32 kcsan_prandom_u32_max(u32 ep_ro)
 {
-	struct rnd_state *state = &get_cpu_var(kcsan_rand_state);
-	const u32 res = prandom_u32_state(state);
+	struct rnd_state *state;
+	u32 res;
+
+	/*
+	 * Avoid recursion with scheduler by using non-tracing versions of
+	 * preempt_disable() and preempt_enable() that do not call into
+	 * scheduler code.
+	 */
+	preempt_disable_notrace();
+	state = raw_cpu_ptr(&kcsan_rand_state);
+	res = prandom_u32_state(state);
+	preempt_enable_no_resched_notrace();
 
-	put_cpu_var(kcsan_rand_state);
 	return (u32)(((u64) res * ep_ro) >> 32);
 }
 
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
maintainer
#1 lore raw
Quoted 43 lines · Marco Elver

When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:

check_access() [via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add() [in kernel/sched/core.c]
check_access() [via instrumentation]

Avoid this by rewriting kcsan_prandom_u32_max() to only use safe versions of preempt_disable() and preempt_enable() that do not call into scheduler code.

Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed from kernel/sched/Makefile to permit testing scheduler code with KCSAN if desired.

Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")

---
v2:
* Update comment to also point out preempt_enable().

---
 kernel/kcsan/core.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
kernel/kcsan/core.c +2 -2
index 3994a217bde7..10513f3e2349 100644
@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
  */
 static u32 kcsan_prandom_u32_max(u32 ep_ro)
 {
-	struct rnd_state *state = &get_cpu_var(kcsan_rand_state);
-	const u32 res = prandom_u32_state(state);
+	struct rnd_state *state;
+	u32 res;

kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298

Quoted 10 lines
kernel/kcsan/core.c +10 -0
+
+	/*
+	 * Avoid recursion with scheduler by using non-tracing versions of
+	 * preempt_disable() and preempt_enable() that do not call into
+	 * scheduler code.
+	 */
+	preempt_disable_notrace();
+	state = raw_cpu_ptr(&kcsan_rand_state);
+	res = prandom_u32_state(state);
+	preempt_enable_no_resched_notrace();

This is a preemption bug. Does preempt_enable_notrace() not work?

Quoted 8 lines
kernel/kcsan/core.c +0 -1
-	put_cpu_var(kcsan_rand_state);
 	return (u32)(((u64) res * ep_ro) >> 32);
 }
 
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
author maintainer
#2 lore raw
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Marco Elver

When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:

check_access() [via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add() [in kernel/sched/core.c]
check_access() [via instrumentation]

Avoid this by rewriting kcsan_prandom_u32_max() to only use safe versions of preempt_disable() and preempt_enable() that do not call into scheduler code.

Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed from kernel/sched/Makefile to permit testing scheduler code with KCSAN if desired.

Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")

---
v2:
* Update comment to also point out preempt_enable().

---
 kernel/kcsan/core.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
kernel/kcsan/core.c +6 -2
index 3994a217bde7..10513f3e2349 100644
@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
  */
 static u32 kcsan_prandom_u32_max(u32 ep_ro)
 {
-     struct rnd_state *state = &get_cpu_var(kcsan_rand_state);
-     const u32 res = prandom_u32_state(state);
+     struct rnd_state *state;
+     u32 res;
+
+     /*
+      * Avoid recursion with scheduler by using non-tracing versions of
+      * preempt_disable() and preempt_enable() that do not call into
Quoted 8 lines

kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298

Quoted 6 lines
kernel/kcsan/core.c +6 -0
+      * scheduler code.
+      */
+     preempt_disable_notrace();
+     state = raw_cpu_ptr(&kcsan_rand_state);
+     res = prandom_u32_state(state);
+     preempt_enable_no_resched_notrace();

This is a preemption bug. Does preempt_enable_notrace() not work?

No it didn't, because we end up calling preempt_schedule_notrace(), which again might end in recursion.

Normally we could surround this by kcsan_disable_current/kcsan_enable_current(), but that doesn't work because we have this sequence:

     reset_kcsan_skip();
     if (!kcsan_is_enabled())
         ...

to avoid underflowing the skip counter if KCSAN is disabled. That could be solved by writing to the skip-counter twice: once with a non-random value, and if KCSAN is enabled with a random value. Would that be better?

And I'd like to avoid adding __no_kcsan to scheduler functions.

Any recommendation?

Thanks,
-- Marco

Quoted 9 lines
Quoted 9 lines
kernel/kcsan/core.c +0 -2
-     put_cpu_var(kcsan_rand_state);
      return (u32)(((u64) res * ep_ro) >> 32);
 }
 
--

2.29.2.454.gaff20da3a2-goog

author maintainer
#3 lore raw
Quoted 65 lines · Marco Elver
Quoted 48 lines · Peter Zijlstra
Quoted 47 lines · Marco Elver

When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe recursion due to:

check_access() [via instrumentation]
kcsan_setup_watchpoint()
reset_kcsan_skip()
kcsan_prandom_u32_max()
get_cpu_var()
preempt_disable()
preempt_count_add() [in kernel/sched/core.c]
check_access() [via instrumentation]

Avoid this by rewriting kcsan_prandom_u32_max() to only use safe versions of preempt_disable() and preempt_enable() that do not call into scheduler code.

Note, while this currently does not affect an unmodified kernel, it'd be good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed from kernel/sched/Makefile to permit testing scheduler code with KCSAN if desired.

Fixes cd290ec24633 ("kcsan: Use tracing-safe version of prandom")

---
v2:
* Update comment to also point out preempt_enable().

---
 kernel/kcsan/core.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
kernel/kcsan/core.c +6 -2
index 3994a217bde7..10513f3e2349 100644
@@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
  */
 static u32 kcsan_prandom_u32_max(u32 ep_ro)
 {
-     struct rnd_state *state = &get_cpu_var(kcsan_rand_state);
-     const u32 res = prandom_u32_state(state);
+     struct rnd_state *state;
+     u32 res;
+
+     /*
+      * Avoid recursion with scheduler by using non-tracing versions of
+      * preempt_disable() and preempt_enable() that do not call into
Quoted 8 lines

kernel/kcsan/core.c should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx * · line 298

Quoted 6 lines
kernel/kcsan/core.c +6 -0
+      * scheduler code.
+      */
+     preempt_disable_notrace();
+     state = raw_cpu_ptr(&kcsan_rand_state);
+     res = prandom_u32_state(state);
+     preempt_enable_no_resched_notrace();

This is a preemption bug. Does preempt_enable_notrace() not work?

No it didn't, because we end up calling preempt_schedule_notrace(), which again might end in recursion.

Normally we could surround this by kcsan_disable_current/kcsan_enable_current(), but that doesn't work because we have this sequence:

Quoted 8 lines
     reset_kcsan_skip();
     if (!kcsan_is_enabled())
         ...

to avoid underflowing the skip counter if KCSAN is disabled. That could be solved by writing to the skip-counter twice: once with a non-random value, and if KCSAN is enabled with a random value. Would that be better?

See below for concrete alternative that works.

Quoted 3 lines

And I'd like to avoid adding __no_kcsan to scheduler functions.

Any recommendation?

Let me know what you prefer.

Thanks,
-- Marco

------ >8 ------

kernel/kcsan/core.c +16 -18
index 10513f3e2349..c8eadef3f42a 100644
@@ -266,8 +266,8 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
 		return false;
 
 	/*
-	 * NOTE: If we get here, kcsan_skip must always be reset in slow path
-	 * via reset_kcsan_skip() to avoid underflow.
+	 * Note: If we get here, kcsan_skip must always be reset in slow path to
+	 * avoid underflow.
 	 */
 
 	/* this operation should be watched */
@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)
 	u32 res;
 
 	/*
-	 * Avoid recursion with scheduler by using non-tracing versions of
-	 * preempt_disable() and preempt_enable() that do not call into
-	 * scheduler code.
+	 * Avoid recursion with scheduler by disabling KCSAN because
+	 * preempt_enable_notrace() will still call into scheduler code.
 	 */
+	kcsan_disable_current();
 	preempt_disable_notrace();
 	state = raw_cpu_ptr(&kcsan_rand_state);
 	res = prandom_u32_state(state);
-	preempt_enable_no_resched_notrace();
+	preempt_enable_notrace();
+	kcsan_enable_current_nowarn();
 
 	return (u32)(((u64) res * ep_ro) >> 32);
 }
 
-static inline void reset_kcsan_skip(void)
-{
-	long skip_count = kcsan_skip_watch -
-			  (IS_ENABLED(CONFIG_KCSAN_SKIP_WATCH_RANDOMIZE) ?
-				   kcsan_prandom_u32_max(kcsan_skip_watch) :
-				   0);
-	this_cpu_write(kcsan_skip, skip_count);
-}
-
 static __always_inline bool kcsan_is_enabled(void)
 {
 	return READ_ONCE(kcsan_enabled) && get_ctx()->disable_count == 0;
@@ -430,10 +422,16 @@ kcsan_setup_watchpoint(const volatile void *ptr, size_t size, int type)
 	 * Always reset kcsan_skip counter in slow-path to avoid underflow; see
 	 * should_watch().
 	 */
-	reset_kcsan_skip();
-
-	if (!kcsan_is_enabled())
+	if (likely(kcsan_is_enabled())) {
+		long skip_count = kcsan_skip_watch -
+				  (IS_ENABLED(CONFIG_KCSAN_SKIP_WATCH_RANDOMIZE) ?
+					   kcsan_prandom_u32_max(kcsan_skip_watch) :
+					   0);
+		this_cpu_write(kcsan_skip, skip_count);
+	} else {
+		this_cpu_write(kcsan_skip, kcsan_skip_watch);
 		goto out;
+	}
 
 	/*
 	 * Special atomic rules: unlikely to be true, so we check them here in
kernel/sched/Makefile +0 -6
index 5fc9c9b70862..21fb5a5662b5 100644
@@ -7,12 +7,6 @@ endif
 # that is not a function of syscall inputs. E.g. involuntary context switches.
 KCOV_INSTRUMENT := n
 
-# There are numerous data races here, however, most of them are due to plain accesses.
-# This would make it even harder for syzbot to find reproducers, because these
-# bugs trigger without specific input. Disable by default, but should re-enable
-# eventually.
-KCSAN_SANITIZE := n
-
 ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
 # needed for x86 only.  Why this used to be enabled for all architectures is beyond
maintainer
#4 lore raw
Quoted 9 lines · Marco Elver

Let me know what you prefer.

kernel/kcsan/core.c +2 -0
@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)
 	u32 res;
 
 	/*
+	 * Avoid recursion with scheduler by disabling KCSAN because
+	 * preempt_enable_notrace() will still call into scheduler code.

kernel/kcsan/core.c static u32 kcsan_prandom_u32_max(u32 ep_ro) · line 302

Quoted 10 lines
kernel/kcsan/core.c +3 -0
 	 */
+	kcsan_disable_current();
 	preempt_disable_notrace();
 	state = raw_cpu_ptr(&kcsan_rand_state);
 	res = prandom_u32_state(state);
+	preempt_enable_notrace();
+	kcsan_enable_current_nowarn();
 
 	return (u32)(((u64) res * ep_ro) >> 32);
 }

This is much preferred over the other. The thing with _no_resched is that you can miss a preemption for an unbounded amount of time, which is bad.

The _only_ valid use of _no_resched is when there's a call to schedule() right after it.

author maintainer
#5 lore raw
Quoted 18 lines · Peter Zijlstra
Quoted 17 lines · Marco Elver

Let me know what you prefer.

kernel/kcsan/core.c +5 -0
@@ -288,27 +288,19 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro)
      u32 res;
 
      /*
+      * Avoid recursion with scheduler by disabling KCSAN because
+      * preempt_enable_notrace() will still call into scheduler code.
       */
+     kcsan_disable_current();
      preempt_disable_notrace();
      state = raw_cpu_ptr(&kcsan_rand_state);
      res = prandom_u32_state(state);
+     preempt_enable_notrace();
+     kcsan_enable_current_nowarn();
 
Quoted 5 lines

kernel/kcsan/core.c static u32 kcsan_prandom_u32_max(u32 ep_ro) · line 302

Quoted 2 lines
kernel/kcsan/core.c +0 -0
      return (u32)(((u64) res * ep_ro) >> 32);
 }

This is much preferred over the other. The thing with _no_resched is that you can miss a preemption for an unbounded amount of time, which is bad.

Ah, I think this is rubbish, too. Because it might fix one problem, but now I'm left with the problem that kcsan_prandom_u32_max() is called for udelay() later and at that point we lose skip_count randomness entirely.

I think relying on lib/random32.c already caused too many headaches, so I'm tempted to just get rid of that dependency entirely. And instead do the simplest possible thing, which might just be calling get_cycles() (all we need is to introduce some non-determinism).

Quoted 2 lines

The _only_ valid use of _no_resched is when there's a call to schedule() right after it.