Skip to contentSkip to conversation

context_tracking: Flatter archs not using exception_enter/exit() v2

PATCH In review Frederic Weisbecker 2 participants 10 messages Oct 27, 2020 – Nov 11, 2020

Subsystems (3)
lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

cover context_tracking: Flatter archs not using exception_enter/exit() v2

In order to be able to turn on/off nohz_full at runtime, we'll need the archs to give up saving the context tracking state on the task stack because this forces context tracking to run system wide, even on CPUs that don't have nohz_full enabled.

Now being able to do so require sane entry code meeting some requirements that at least x86 just achieved recently (I haven't checked other archs yet).

Changes since v2:

- Rebase against v5.10-rc1
- Use SCHED_WARN_ON() instead of WARN_ON_ONCE() for the context tracking sanity check on schedule().

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
core/isolation-v2

HEAD: 79f60f3dd0e0aea8b17c825371d8697444ae5faf

Thanks,
Frederic
---

Frederic Weisbecker (5):
context_tracking: Introduce HAVE_CONTEXT_TRACKING_OFFSTACK
context_tracking: Don't implement exception_enter/exit() on CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK
sched: Detect call to schedule from critical entry code
context_tracking: Only define schedule_user() on !HAVE_CONTEXT_TRACKING_OFFSTACK archs
x86: Support HAVE_CONTEXT_TRACKING_OFFSTACK

 arch/Kconfig                     | 17 +++++++++++++++++
 arch/x86/Kconfig                 |  1 +
 include/linux/context_tracking.h |  6 ++++--
 kernel/sched/core.c              |  3 ++-
 4 files changed, 24 insertions(+), 3 deletions(-)
author maintainer
#1 lore raw

PATCH 1/5 context_tracking: Introduce HAVE_CONTEXT_TRACKING_OFFSTACK

Historically, context tracking had to deal with fragile entry code path,
ie: before user_exit() is called and after user_enter() is called, in case some of those spots would call schedule() or use RCU. On such cases, the site had to be protected between exception_enter() and exception_exit() that save the context tracking state in the task stack.

Such sleepable fragile code path had many different origins: tracing, exceptions, early or late calls to context tracking on syscalls...

Aside of that not being pretty, saving the context tracking state on the task stack forces us to run context tracking on all CPUs, including housekeepers, and prevents us to completely shutdown nohz_full at runtime on a CPU in the future as context tracking and its overhead would still need to run system wide.

Now thanks to the extensive efforts to sanitize x86 entry code, those conditions have been removed and we can now get rid of these workarounds in this architecture.

Create a Kconfig feature to express this achievement.

Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/Kconfig | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/Kconfig +17 -0
index 56b6ccc0e32d..090ef3566c56 100644
@@ -618,6 +618,23 @@ config HAVE_CONTEXT_TRACKING
 	  protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal
 	  handling on irq exit still need to be protected.
 
+config HAVE_CONTEXT_TRACKING_OFFSTACK
+	bool
+	help
+	  Architecture neither relies on exception_enter()/exception_exit()
+	  nor on schedule_user(). Also preempt_schedule_notrace() and
+	  preempt_schedule_irq() can't be called in a preemptible section
+	  while context tracking is CONTEXT_USER. This feature reflects a sane
+	  entry implementation where the following requirements are met on
+	  critical entry code, ie: before user_exit() or after user_enter():
+
+	  - Critical entry code isn't preemptible (or better yet:
+	    not interruptible).
+	  - No use of RCU read side critical sections, unless rcu_nmi_enter()
+	    got called.
+	  - No use of instrumentation, unless instrumentation_begin() got
+	    called.
+
 config HAVE_TIF_NOHZ
 	bool
 	help
Signature · 1 line
2.25.1
author maintainer
#2 lore raw

PATCH 2/5 context_tracking: Don't implement exception_enter/exit() on CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK

In reply to Frederic Weisbecker

An architecture that provides this Kconfig feature doesn't need to store the context tracking state on the task stack because its entry code has been sanitized such that fragile path aren't preemptible and special use of tracing and RCU read side critical sections in these areas have been explicitly annotated.

Hence the exception_enter()/exception_exit() couple doesn't need to be implemented in this case.

Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/context_tracking.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
include/linux/context_tracking.h +4 -2
index d53cd331c4dd..bceb06498521 100644
@@ -51,7 +51,8 @@ static inline enum ctx_state exception_enter(void)
 {
 	enum ctx_state prev_ctx;
 
-	if (!context_tracking_enabled())
+	if (IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) ||
+	    !context_tracking_enabled())
 		return 0;
 
 	prev_ctx = this_cpu_read(context_tracking.state);
@@ -63,7 +64,8 @@ static inline enum ctx_state exception_enter(void)
 
 static inline void exception_exit(enum ctx_state prev_ctx)
 {
-	if (context_tracking_enabled()) {
+	if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) &&
+	    context_tracking_enabled()) {
 		if (prev_ctx != CONTEXT_KERNEL)
 			context_tracking_enter(prev_ctx);
 	}
Signature · 1 line
2.25.1
maintainer
#3 lore raw

Re: [PATCH 2/5] context_tracking: Don't implement exception_enter/exit() on CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK

Quoted 8 lines · Frederic Weisbecker

An architecture that provides this Kconfig feature doesn't need to store the context tracking state on the task stack because its entry code has been sanitized such that fragile path aren't preemptible and special use of tracing and RCU read side critical sections in these areas have been explicitly annotated.

Hence the exception_enter()/exception_exit() couple doesn't need to be implemented in this case.

Could you please explain what exception_{enter,exit}() actually do, then explain what is required to make it superfluous? Because as is, I don't have enough information to verify the claims made.

Quoted 37 lines
Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/context_tracking.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
include/linux/context_tracking.h +4 -2
index d53cd331c4dd..bceb06498521 100644
@@ -51,7 +51,8 @@ static inline enum ctx_state exception_enter(void)
 {
 	enum ctx_state prev_ctx;
 
-	if (!context_tracking_enabled())
+	if (IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) ||
+	    !context_tracking_enabled())
 		return 0;
 
 	prev_ctx = this_cpu_read(context_tracking.state);
@@ -63,7 +64,8 @@ static inline enum ctx_state exception_enter(void)
 
 static inline void exception_exit(enum ctx_state prev_ctx)
 {
-	if (context_tracking_enabled()) {
+	if (!IS_ENABLED(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK) &&
+	    context_tracking_enabled()) {
 		if (prev_ctx != CONTEXT_KERNEL)
 			context_tracking_enter(prev_ctx);
 	}
Signature · 1 line
2.25.1
author maintainer
#4 lore raw

Re: [PATCH 2/5] context_tracking: Don't implement exception_enter/exit() on CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK

Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Frederic Weisbecker

An architecture that provides this Kconfig feature doesn't need to store the context tracking state on the task stack because its entry code has been sanitized such that fragile path aren't preemptible and special use of tracing and RCU read side critical sections in these areas have been explicitly annotated.

Quoted 6 lines
Quoted 2 lines

Hence the exception_enter()/exception_exit() couple doesn't need to be implemented in this case.

Could you please explain what exception_{enter,exit}() actually do, then explain what is required to make it superfluous? Because as is, I don't have enough information to verify the claims made.

So, the typical steps with context tracking are:

1) Task runs in userspace
2) Task enters the kernel (syscall/exception/IRQ)
3) Task switches from context tracking state CONTEXT_USER to CONTEXT_KERNEL (user_exit())
4) Task does stuff in kernel
5) Task switches from context tracking state CONTEXT_KERNEL to CONTEXT_USER (user_enter())
6) Task exits the kernel

If an exception fires between 5) and 6), the pt_regs and the context tracking disagree on the context of the origin. We must set CONTEXT_KERNEL before the exception handler, that's unconditional for those that want to call into schedule, but we must restore CONTEXT_USER when we exit the exception while pt_regs tells us that we are resuming to kernel space.

We can't fix that with storing the context tracking state in a per-cpu or per-task variable since another exception may fire onto the current one and overwrite the saved state. Also the task can schedule. So it has to be stored in a per task stack.

This is how exception_enter()/exception_exit() papers over the problem:

5) Task switches from context tracking state CONTEXT_KERNEL to CONTEXT_USER (user_enter()) 5.1) Exception fires
5.2) prev_state = exception_enter() // save CONTEXT_USER to prev_state and set CONTEXT_KERNEL 5.3) Exception handler
5.4) exception_enter(prev_state) // restore CONTEXT_USER
5.5) Exception resumes
6) Task exits the kernel

What is required to make it superfluous is to forbid exceptions and IRQs between 2) and 3) and between 5) and 6), or if we allow any, they won't call into context tracking, eg: NMIs, and they won't schedule.

I can add that in the changelog if that makes it any clearer.

Thanks.

author maintainer
#5 lore raw

PATCH 3/5 sched: Detect call to schedule from critical entry code

In reply to Frederic Weisbecker

Detect calls to schedule() between user_enter() and user_exit(). Those are symptoms of early entry code that either forgot to protect a call to schedule() inside exception_enter()/exception_exit() or, in the case of HAVE_CONTEXT_TRACKING_OFFSTACK, enabled interrupts or preemption in a wrong spot.

Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/sched/core.c | 1 +
 1 file changed, 1 insertion(+)
kernel/sched/core.c +1 -0
index d2003a7d5ab5..c23d7cb5aee3 100644
@@ -4291,6 +4291,7 @@ static inline void schedule_debug(struct task_struct *prev, bool preempt)
 		preempt_count_set(PREEMPT_DISABLED);
 	}
 	rcu_sleep_check();
+	SCHED_WARN_ON(ct_state() == CONTEXT_USER);
 
 	profile_hit(SCHED_PROFILING, __builtin_return_address(0));
 
Signature · 1 line
2.25.1
author maintainer
#6 lore raw

PATCH 4/5 context_tracking: Only define schedule_user() on !HAVE_CONTEXT_TRACKING_OFFSTACK archs

In reply to Frederic Weisbecker

schedule_user() was traditionally used by the entry code's tail to preempt userspace after the call to user_enter(). Indeed the call to user_enter() used to be performed upon syscall exit slow path which was right before the last opportunity to schedule() while resuming to userspace. The context tracking state had to be saved on the task stack and set back to CONTEXT_KERNEL temporarily in order to safely switch to another task.

Only a few archs use it now and those implementing HAVE_CONTEXT_TRACKING_OFFSTACK definetly can't rely on it.

Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index c23d7cb5aee3..44426e5acde2 100644
@@ -4631,7 +4631,7 @@ void __sched schedule_idle(void)
 	} while (need_resched());
 }
 
-#ifdef CONFIG_CONTEXT_TRACKING
+#if defined(CONFIG_CONTEXT_TRACKING) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK)
 asmlinkage __visible void __sched schedule_user(void)
 {
 	/*
Signature · 1 line
2.25.1
maintainer
#7 lore raw

Re: [PATCH 4/5] context_tracking: Only define schedule_user() on !HAVE_CONTEXT_TRACKING_OFFSTACK archs

Quoted 10 lines · Frederic Weisbecker

schedule_user() was traditionally used by the entry code's tail to preempt userspace after the call to user_enter(). Indeed the call to user_enter() used to be performed upon syscall exit slow path which was right before the last opportunity to schedule() while resuming to userspace. The context tracking state had to be saved on the task stack and set back to CONTEXT_KERNEL temporarily in order to safely switch to another task.

Only a few archs use it now and those implementing HAVE_CONTEXT_TRACKING_OFFSTACK definetly can't rely on it.

powerpc64 and sparc64 only, right?

Quoted 26 lines
Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index c23d7cb5aee3..44426e5acde2 100644
@@ -4631,7 +4631,7 @@ void __sched schedule_idle(void)
 	} while (need_resched());
 }
 
-#ifdef CONFIG_CONTEXT_TRACKING
+#if defined(CONFIG_CONTEXT_TRACKING) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK)
 asmlinkage __visible void __sched schedule_user(void)
 {
 	/*
Signature · 1 line
2.25.1
author maintainer
#8 lore raw

Re: [PATCH 4/5] context_tracking: Only define schedule_user() on !HAVE_CONTEXT_TRACKING_OFFSTACK archs

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Frederic Weisbecker

schedule_user() was traditionally used by the entry code's tail to preempt userspace after the call to user_enter(). Indeed the call to user_enter() used to be performed upon syscall exit slow path which was right before the last opportunity to schedule() while resuming to userspace. The context tracking state had to be saved on the task stack and set back to CONTEXT_KERNEL temporarily in order to safely switch to another task.

Quoted 4 lines
Quoted 2 lines

Only a few archs use it now and those implementing HAVE_CONTEXT_TRACKING_OFFSTACK definetly can't rely on it.

powerpc64 and sparc64 only, right?

Exactly! I can precise that in the changelog.

Thanks.

Quoted 27 lines
Quoted 27 lines
Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index c23d7cb5aee3..44426e5acde2 100644
@@ -4631,7 +4631,7 @@ void __sched schedule_idle(void)
 	} while (need_resched());
 }
 
-#ifdef CONFIG_CONTEXT_TRACKING
+#if defined(CONFIG_CONTEXT_TRACKING) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK)
 asmlinkage __visible void __sched schedule_user(void)
 {
 	/*
Signature · 1 line
2.25.1
author maintainer
#9 lore raw

PATCH 5/5 x86: Support HAVE_CONTEXT_TRACKING_OFFSTACK

In reply to Frederic Weisbecker

A lot of ground work has been performed on x86 entry code. Fragile path between user_enter() and user_exit() have IRQs disabled. Uses of RCU and intrumentation in these fragile areas have been explicitly annotated and protected.

This architecture doesn't need exception_enter()/exception_exit() anymore and has therefore earned CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK.

Cc Marcelo Tosatti <[email protected]>
Cc Paul E. McKenney <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Phil Auld <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 arch/x86/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/x86/Kconfig +1 -0
index f6946b81f74a..d793361839b5 100644
@@ -162,6 +162,7 @@ config X86
 	select HAVE_CMPXCHG_DOUBLE
 	select HAVE_CMPXCHG_LOCAL
 	select HAVE_CONTEXT_TRACKING		if X86_64
+	select HAVE_CONTEXT_TRACKING_OFFSTACK	if HAVE_CONTEXT_TRACKING
 	select HAVE_C_RECORDMCOUNT
 	select HAVE_DEBUG_KMEMLEAK
 	select HAVE_DMA_CONTIGUOUS
Signature · 1 line
2.25.1