idle-related changes
Subsystems (10)
cover idle-related changes
Hello!
This series handles changes to the interaction between RCU and idle, including adaptive ticks. Almost all of these patches are courtesy of Frederic Weisbecker. The patches are as follows:
1. Allow non-idle tasks to enter dyntick-idle mode from RCU's
perspective in order to enable adaptive ticks.
2. Add rcu_user_enter_irq() and rcu_user_exit_irq() to enable
switching into and out of dyntick-idle mode in interrupt handlers to handle wakeups.
3. Modify RCU_FAST_NO_HZ to accommodate adaptive ticks.
4. Add a Kconfig option to enable extended quiescent states while
in usermode execution.
5. Allow multiple rcu_user_enter() calls to match an rcu_user_exit().
6. Introcude runtime control for RCU's classification of user-mode
execution as an extended quiescent state.
7. Update adaptive-tick state on context switch in order to handle
task migration.
8. On x86, notify RCU of slowpath syscall entry to and exit from
usermode execution so that RCU can track the resulting extended quiescent states.
9. On c86, notify RCU of exception-path entry to and exit from
usermode execution so that RCU can again strack the resulting extended quiescent states.
10. Exit RCU extended QS on kernel preemption after irq/exception.
11. Exit RCU extended QS on user preemption.
12. On x86, use the new schedule_user API on userspace preemption.
13. On x86, exit RCU extended QS on notify resumes.
14. Provide a new RCU_USER_QS_FORCE kconfig option that enables
userspace RCU extended quiescent states on all CPUs for testing purposes.
15-26. Fix idle-loop breakage introduced on 3.3. This affects all
architectures that do not implement NO_HZ.
Thanx, Paul
------------------------------------------------------------------------
arch/alpha/kernel/process.c | 3 b/arch/Kconfig | 10 +++ b/arch/alpha/kernel/process.c | 3 b/arch/alpha/kernel/smp.c | 1 b/arch/cris/kernel/process.c | 3 b/arch/frv/kernel/process.c | 3 b/arch/h8300/kernel/process.c | 3 b/arch/ia64/kernel/process.c | 3 b/arch/m32r/kernel/process.c | 3 b/arch/m68k/kernel/process.c | 3 b/arch/mn10300/kernel/process.c | 3 b/arch/parisc/kernel/process.c | 3 b/arch/score/kernel/process.c | 4 - b/arch/um/drivers/mconsole_kern.c | 1 b/arch/x86/Kconfig | 1 b/arch/x86/include/asm/rcu.h | 20 ++++++ b/arch/x86/include/asm/thread_info.h | 10 ++- b/arch/x86/kernel/entry_64.S | 8 +- b/arch/x86/kernel/ptrace.c | 5 + b/arch/x86/kernel/signal.c | 4 + b/arch/x86/kernel/traps.c | 30 ++++++--- b/arch/x86/mm/fault.c | 13 +++ b/arch/xtensa/kernel/process.c | 3 b/include/linux/rcupdate.h | 2 b/include/linux/sched.h | 8 ++ b/init/Kconfig | 10 +++ b/kernel/rcutree.c | 115 ++++++++++++++++++++++++----------- b/kernel/rcutree.h | 3 b/kernel/rcutree_plugin.h | 20 ++++++ b/kernel/sched/core.c | 1 include/linux/rcupdate.h | 12 +++ init/Kconfig | 8 ++ kernel/rcutree.c | 111 ++++++++++++++++++++++++++++++--- kernel/rcutree.h | 1 kernel/sched/core.c | 8 ++ 35 files changed, 375 insertions(+), 64 deletions(-)
PATCH 1/26 rcu: New rcu_user_enter() and rcu_user_exit() APIs
From: Frederic Weisbecker <[email protected]>
RCU currently insists that only idle tasks can enter RCU idle mode, which prohibits an adaptive tickless kernel (AKA nohz cpusets), which in turn would mean that usermode execution would always take scheduling-clock interrupts, even when there is only one task runnable on the CPU in question.
This commit therefore adds rcu_user_enter() and rcu_user_exit(), which allow non-idle tasks to enter RCU idle mode. These are quite similar to rcu_idle_enter() and rcu_idle_exit(), respectively, except that they omit the idle-task checks.
[ Updated to use "user" flag rather than separate check functions. ]
--- include/linux/rcupdate.h | 2 + kernel/rcutree.c | 115 ++++++++++++++++++++++++++++++++-------------- 2 files changed, 83 insertions(+), 34 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -191,6 +191,8 @@ extern void rcu_idle_enter(void);extern void rcu_idle_exit (void );extern void rcu_irq_enter (void );extern void rcu_irq_exit (void ); +extern void rcu_user_enter (void ); +extern void rcu_user_exit (void );extern void exit_rcu (void );/**
kernel/rcutree.c +81 -34
197 lines not on this page · raw message
Signature · 1 line
1.7.8
PATCH 2/26 rcu: New rcu_user_enter_irq() and rcu_user_exit_irq() APIs
From: Frederic Weisbecker <[email protected]>
In some cases, it is necessary to enter or exit userspace-RCU-idle mode from an interrupt handler, for example, if some other CPU sends this CPU a resched IPI. In this case, the current CPU would enter the IPI handler in userspace-RCU-idle mode, but would need to exit the IPI handler after having exited that mode.
To allow this to work, this commit adds two new APIs to TREE_RCU:
- rcu_user_enter_irq(). This must be called from an interrupt between rcu_irq_enter() and rcu_irq_exit(). After the irq calls rcu_irq_exit(), the irq handler will return into an RCU extended quiescent state. In theory, this interrupt is never a nested interrupt, but in practice it might interrupt softirq, which looks to RCU like a nested interrupt.
- rcu_user_exit_irq(). This must be called from a non-nesting interrupt, interrupting an RCU extended quiescent state, also between rcu_irq_enter() and rcu_irq_exit(). After the irq calls rcu_irq_exit(), the irq handler will return in an RCU non-quiescent state.
[ Combined with "Allow calls to rcu_exit_user_irq from nesting irqs." ]
--- include/linux/rcupdate.h | 2 ++ kernel/rcutree.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 0 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -193,6 +193,8 @@ extern void rcu_irq_enter(void);extern void rcu_irq_exit (void );extern void rcu_user_enter (void );extern void rcu_user_exit (void ); +extern void rcu_user_enter_irq (void ); +extern void rcu_user_exit_irq (void );extern void exit_rcu (void );/**
kernel/rcutree.c +43 -0
55 lines not on this page · raw message
Signature · 1 line
1.7.8
Josh Triplett
maintainerRe: [PATCH tip/core/rcu 02/26] rcu: New rcu_user_enter_irq() and rcu_user_exit_irq() APIs
Quoted 16 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In some cases, it is necessary to enter or exit userspace-RCU-idle mode from an interrupt handler, for example, if some other CPU sends this CPU a resched IPI. In this case, the current CPU would enter the IPI handler in userspace-RCU-idle mode, but would need to exit the IPI handler after having exited that mode.
To allow this to work, this commit adds two new APIs to
TREE_RCU:-
rcu_user_enter_irq(). This must be called from an interrupt betweenrcu_irq_enter()andrcu_irq_exit(). After the irq callsrcu_irq_exit(), the irq handler will return into an RCU extended quiescent state. In theory, this interrupt is never a nested interrupt, but in practice it might interrupt softirq, which looks to RCU like a nested interrupt.
Quoted 5 lines
-
rcu_user_exit_irq(). This must be called from a non-nesting interrupt, interrupting an RCU extended quiescent state, also betweenrcu_irq_enter()andrcu_irq_exit(). After the irq callsrcu_irq_exit(), the irq handler will return in an RCU non-quiescent state.
These names seem a bit confusing. From the descriptions, it sounds like you don't always need to pair them; rcu_irq_exit() will return to a non-quiescent state, unless you call rcu_user_enter_irq and *don't* call rcu_user_exit_irq. Did I get that semantic right?
Given that, the "enter" and "exit" names seem confusing. This seems more like a flag you can set and clear, rather than a delimited region as suggested by an enter/exit pair.
How about something vaguely like rcu_user_irq_set_eqs and rcu_user_irq_clear_eqs?
- Josh Triplett
Re: [PATCH tip/core/rcu 02/26] rcu: New rcu_user_enter_irq() and rcu_user_exit_irq() APIs
2012/8/31 Josh Triplett <[email protected]>:
Quoted 23 lines · Paul E. McKenney
Quoted 16 lines
From: Frederic Weisbecker <[email protected]>
In some cases, it is necessary to enter or exit userspace-RCU-idle mode from an interrupt handler, for example, if some other CPU sends this CPU a resched IPI. In this case, the current CPU would enter the IPI handler in userspace-RCU-idle mode, but would need to exit the IPI handler after having exited that mode.
To allow this to work, this commit adds two new APIs to
TREE_RCU:-
rcu_user_enter_irq(). This must be called from an interrupt betweenrcu_irq_enter()andrcu_irq_exit(). After the irq callsrcu_irq_exit(), the irq handler will return into an RCU extended quiescent state. In theory, this interrupt is never a nested interrupt, but in practice it might interrupt softirq, which looks to RCU like a nested interrupt.Quoted 5 lines
-
rcu_user_exit_irq(). This must be called from a non-nesting interrupt, interrupting an RCU extended quiescent state, also betweenrcu_irq_enter()andrcu_irq_exit(). After the irq callsrcu_irq_exit(), the irq handler will return in an RCU non-quiescent state.
Quoted 4 lines
These names seem a bit confusing. From the descriptions, it sounds like you don't always need to pair them;
rcu_irq_exit()will return to a non-quiescent state, unless you callrcu_user_enter_irqand *don't* callrcu_user_exit_irq. Did I get that semantic right?
Yeah. They indeed don't always need to be paired. We can enter into user (from rcu POV) with rcu_user_enter_irq() and exit user with rcu_user_exit().
It's just a matter of context: from where do we set/unset RCU as in user mode: irq or not. The only thing that is paired is the fact we enter/exit that RCU user mode. There are just different APIs to do so.
Quoted 6 lines
Given that, the "enter" and "exit" names seem confusing. This seems more like a flag you can set and clear, rather than a delimited region as suggested by an enter/exit pair.
How about something vaguely like
rcu_user_irq_set_eqsandrcu_user_irq_clear_eqs?
I'd rather suggest rcu_user_enter_after_irq and rcu_user_exit_after_irq. It describes precisely what it does.
Quoted 2 lines
- Josh Triplett
Josh Triplett
maintainerRe: [PATCH tip/core/rcu 02/26] rcu: New rcu_user_enter_irq() and rcu_user_exit_irq() APIs
Quoted 10 lines · Frederic Weisbecker
2012/8/31 Josh Triplett <[email protected]>:
Quoted 6 lines
Given that, the "enter" and "exit" names seem confusing. This seems more like a flag you can set and clear, rather than a delimited region as suggested by an enter/exit pair.
How about something vaguely like
rcu_user_irq_set_eqsandrcu_user_irq_clear_eqs?I'd rather suggest
rcu_user_enter_after_irqandrcu_user_exit_after_irq. It describes precisely what it does.
Those names sound reasonable, sure; in the context of "after", enter/exit sounds less confusing.
- Josh Triplett
PATCH 3/26 rcu: Make RCU_FAST_NO_HZ handle adaptive ticks
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The current implementation of RCU_FAST_NO_HZ tries reasonably hard to rid the current CPU of RCU callbacks. This is appropriate when the CPU is entering idle, where it doesn't have much useful to do anyway, but is most definitely not what you want when transitioning to user-mode execution. This commit therefore detects the adaptive-tick case, and refrains from burning CPU time getting rid of RCU callbacks in that case.
--- kernel/rcutree_plugin.h | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-)
kernel/rcutree_plugin.h +20 -0
@@ -1997,6 +1997,26 @@ static void rcu_prepare_for_idle(int cpu)if (!tne)return ; +/* Adaptive-tick mode, where usermode execution is idle to RCU. */ +if (!is_idle_task (current)) { + rdtp->dyntick_holdoff = jiffies -1 ; +if (rcu_cpu_has_nonlazy_callbacks (cpu)) { +trace_rcu_prep_idle ("User dyntick with callbacks" ); + rdtp->idle_gp_timer_expires = +round_up (jiffies +RCU_IDLE_GP_DELAY , +RCU_IDLE_GP_DELAY ); + }else if (rcu_cpu_has_callbacks (cpu)) { + rdtp->idle_gp_timer_expires = +round_jiffies (jiffies +RCU_IDLE_LAZY_GP_DELAY ); +trace_rcu_prep_idle ("User dyntick with lazy callbacks" ); + }else { +return ; + } + tp = &rdtp->idle_gp_timer; +mod_timer_pinned (tp, rdtp->idle_gp_timer_expires); +return ; + } +/* * If this is an idle re-entry, for example, due to use of * RCU_NONIDLE() or the new idle-loop tracing API within the idle
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 03/26] rcu: Make RCU_FAST_NO_HZ handle adaptive ticks
Quoted 8 lines · Paul E. McKenney
From: "Paul E. McKenney" <[email protected]>
The current implementation of
RCU_FAST_NO_HZtries reasonably hard to rid the current CPU of RCU callbacks. This is appropriate when the CPU is entering idle, where it doesn't have much useful to do anyway, but is most definitely not what you want when transitioning to user-mode execution. This commit therefore detects the adaptive-tick case, and refrains from burning CPU time getting rid of RCU callbacks in that case.
With the OOM handler from your other patch series, I don't know that it makes as much sense in the idle case, either; perhaps it would make more sense to wait and batch up more callbacks as long as you have memory, and then run them in one big burst.
Quoted 2 lines
Signed-off-by Paul E. McKenney <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 37 lines
kernel/rcutree_plugin.h | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-)kernel/rcutree_plugin.h +20
-0@@ -1997,6 +1997,26 @@ static void rcu_prepare_for_idle(int cpu)if (!tne)return ; +/* Adaptive-tick mode, where usermode execution is idle to RCU. */ +if (!is_idle_task (current)) { + rdtp->dyntick_holdoff = jiffies -1 ; +if (rcu_cpu_has_nonlazy_callbacks (cpu)) { +trace_rcu_prep_idle ("User dyntick with callbacks" ); + rdtp->idle_gp_timer_expires = +round_up (jiffies +RCU_IDLE_GP_DELAY , +RCU_IDLE_GP_DELAY ); + }else if (rcu_cpu_has_callbacks (cpu)) { + rdtp->idle_gp_timer_expires = +round_jiffies (jiffies +RCU_IDLE_LAZY_GP_DELAY ); +trace_rcu_prep_idle ("User dyntick with lazy callbacks" ); + }else { +return ; + } + tp = &rdtp->idle_gp_timer; +mod_timer_pinned (tp, rdtp->idle_gp_timer_expires); +return ; + } +/* * If this is an idle re-entry, for example, due to use of * RCU_NONIDLE() or the new idle-loop tracing API within the idle Signature · 1 line
1.7.8
PATCH 4/26 rcu: Settle config for userspace extended quiescent state
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Create a new config option under the RCU menu that put CPUs under RCU extended quiescent state (as in dynticks idle mode) when they run in userspace. This require some contribution from architectures to hook into kernel and userspace boundaries.
--- arch/Kconfig | 10 ++++++++++ include/linux/rcupdate.h | 8 ++++++++ init/Kconfig | 10 ++++++++++ kernel/rcutree.c | 5 ++++- 4 files changed, 32 insertions(+), 1 deletions(-)
arch/Kconfig +10 -0
@@ -281,4 +281,14 @@ config SECCOMP_FILTER See Documentation/prctl/seccomp_filter.txt for details. +config HAVE_RCU_USER_QS + bool + help + Provide kernel entry/exit hooks necessary for userspace + RCU extended quiescent state. Syscalls need to be wrapped inside + rcu_user_exit()-rcu_user_enter() through the slow path using + TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs + are already protected inside rcu_irq_enter/rcu_irq_exit() but + preemption or signal handling on irq exit still need to be protected. + source "kernel/gcov/Kconfig"
include/linux/rcupdate.h +8 -0
@@ -191,10 +191,18 @@ extern void rcu_idle_enter(void);extern void rcu_idle_exit (void );extern void rcu_irq_enter (void );extern void rcu_irq_exit (void ); + +#ifdef CONFIG_RCU_USER_QS extern void rcu_user_enter (void );extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +#else +static inline void rcu_user_enter (void ) { } +static inline void rcu_user_exit (void ) { } +#endif /* CONFIG_RCU_USER_QS */ + +extern void exit_rcu (void );/**
init/Kconfig +10 -0
@@ -441,6 +441,16 @@ config PREEMPT_RCU This option enables preemptible-RCU code that is common between the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. +config RCU_USER_QS + bool "Consider userspace as in RCU extended quiescent state" + depends on HAVE_RCU_USER_QS && SMP + help + This option sets hooks on kernel / userspace boundaries and + puts RCU in extended quiescent state when the CPU runs in + userspace. It means that when a CPU runs in userspace, it is + excluded from the global RCU state machine and thus doesn't + to keep the timer tick on for RCU. + config RCU_FANOUT int "Tree-based hierarchical RCU fanout value" range 2 64 if 64BIT
kernel/rcutree.c +4 -1
@@ -424,6 +424,7 @@ void rcu_idle_enter(void) }EXPORT_SYMBOL_GPL (rcu_idle_enter); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_enter - inform RCU that we are resuming userspace. *
@@ -438,7 +439,6 @@ void rcu_user_enter(void) }EXPORT_SYMBOL_GPL (rcu_user_enter); -/** * rcu_user_enter_irq - inform RCU that we are going to resume userspace * after the current irq returns.
@@ -459,6 +459,7 @@ void rcu_user_enter_irq(void) rdtp->dynticks_nesting =1 ; local_irq_restore(flags); } +#endif /** * rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle
@@ -562,6 +563,7 @@ void rcu_idle_exit(void) }EXPORT_SYMBOL_GPL (rcu_idle_exit); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_exit - inform RCU that we are exiting userspace. *
@@ -595,6 +597,7 @@ void rcu_user_exit_irq(void) rdtp->dynticks_nesting +=DYNTICK_TASK_EXIT_IDLE ; local_irq_restore(flags); } +#endif /** * rcu_irq_enter - inform RCU that current CPU is entering irq away from idle
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 04/26] rcu: Settle config for userspace extended quiescent state
Quoted 20 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Create a new config option under the RCU menu that put CPUs under RCU extended quiescent state (as in dynticks idle mode) when they run in userspace. This require some contribution from architectures to hook into kernel and userspace boundaries.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
One question below, but nonethelesss:
Quoted 53 lines
arch/Kconfig | 10 ++++++++++ include/linux/rcupdate.h | 8 ++++++++ init/Kconfig | 10 ++++++++++ kernel/rcutree.c | 5 ++++- 4 files changed, 32 insertions(+), 1 deletions(-)arch/Kconfig +10
-0@@ -281,4 +281,14 @@ config SECCOMP_FILTER See Documentation/prctl/seccomp_filter.txt for details. +config HAVE_RCU_USER_QS + bool + help + Provide kernel entry/exit hooks necessary for userspace + RCU extended quiescent state. Syscalls need to be wrapped inside + rcu_user_exit()-rcu_user_enter() through the slow path using + TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs + are already protected inside rcu_irq_enter/rcu_irq_exit() but + preemption or signal handling on irq exit still need to be protected. + source "kernel/gcov/Kconfig"include/linux/rcupdate.h +8
-0@@ -191,10 +191,18 @@ extern void rcu_idle_enter(void);extern void rcu_idle_exit (void );extern void rcu_irq_enter (void );extern void rcu_irq_exit (void ); + +#ifdef CONFIG_RCU_USER_QS extern void rcu_user_enter (void );extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +#else +static inline void rcu_user_enter (void ) { } +static inline void rcu_user_exit (void ) { } +#endif /* CONFIG_RCU_USER_QS */ + +extern void exit_rcu (void );/** init/Kconfig +0
-0
init/Kconfig config PREEMPT_RCU · line 446
Quoted 6 lines
init/Kconfig +3
-0@@ -441,6 +441,16 @@ config PREEMPT_RCU This option enables preemptible-RCU code that is common between the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. +config RCU_USER_QS + bool "Consider userspace as in RCU extended quiescent state" + depends on HAVE_RCU_USER_QS && SMP
Does this actually depend on SMP, or does it depend on the non-TINY RCU implementation? If the latter, it should depend on that rather than SMP.
(I assume that the tiny RCU implementation simply doesn't need all this machinery because it doesn't need coordinated quiescence at all? Or does tiny RCU still cause a periodic wakeup on UP?)
Quoted 57 lines
init/Kconfig +7
-0+ help + This option sets hooks on kernel / userspace boundaries and + puts RCU in extended quiescent state when the CPU runs in + userspace. It means that when a CPU runs in userspace, it is + excluded from the global RCU state machine and thus doesn't + to keep the timer tick on for RCU. + config RCU_FANOUT int "Tree-based hierarchical RCU fanout value" range 2 64 if 64BITkernel/rcutree.c +4
-1@@ -424,6 +424,7 @@ void rcu_idle_enter(void) }EXPORT_SYMBOL_GPL (rcu_idle_enter); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_enter - inform RCU that we are resuming userspace. * @@ -438,7 +439,6 @@ void rcu_user_enter(void) }EXPORT_SYMBOL_GPL (rcu_user_enter); -/** * rcu_user_enter_irq - inform RCU that we are going to resume userspace * after the current irq returns. @@ -459,6 +459,7 @@ void rcu_user_enter_irq(void) rdtp->dynticks_nesting =1 ; local_irq_restore(flags); } +#endif /** * rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle @@ -562,6 +563,7 @@ void rcu_idle_exit(void) }EXPORT_SYMBOL_GPL (rcu_idle_exit); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_exit - inform RCU that we are exiting userspace. * @@ -595,6 +597,7 @@ void rcu_user_exit_irq(void) rdtp->dynticks_nesting +=DYNTICK_TASK_EXIT_IDLE ; local_irq_restore(flags); } +#endif /** * rcu_irq_enter - inform RCU that current CPU is entering irq away from idle Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 04/26] rcu: Settle config for userspace extended quiescent state
Quoted 98 lines · Josh Triplett
Quoted 20 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Create a new config option under the RCU menu that put CPUs under RCU extended quiescent state (as in dynticks idle mode) when they run in userspace. This require some contribution from architectures to hook into kernel and userspace boundaries.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>One question below, but nonethelesss:
Reviewed-by Josh Triplett <[email protected]>Quoted 53 lines
arch/Kconfig | 10 ++++++++++ include/linux/rcupdate.h | 8 ++++++++ init/Kconfig | 10 ++++++++++ kernel/rcutree.c | 5 ++++- 4 files changed, 32 insertions(+), 1 deletions(-)arch/Kconfig +10
-0@@ -281,4 +281,14 @@ config SECCOMP_FILTER See Documentation/prctl/seccomp_filter.txt for details. +config HAVE_RCU_USER_QS + bool + help + Provide kernel entry/exit hooks necessary for userspace + RCU extended quiescent state. Syscalls need to be wrapped inside + rcu_user_exit()-rcu_user_enter() through the slow path using + TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs + are already protected inside rcu_irq_enter/rcu_irq_exit() but + preemption or signal handling on irq exit still need to be protected. + source "kernel/gcov/Kconfig"include/linux/rcupdate.h +8
-0@@ -191,10 +191,18 @@ extern void rcu_idle_enter(void);extern void rcu_idle_exit (void );extern void rcu_irq_enter (void );extern void rcu_irq_exit (void ); + +#ifdef CONFIG_RCU_USER_QS extern void rcu_user_enter (void );extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +#else +static inline void rcu_user_enter (void ) { } +static inline void rcu_user_exit (void ) { } +#endif /* CONFIG_RCU_USER_QS */ + +extern void exit_rcu (void );/** init/Kconfig +0
-0init/Kconfig config PREEMPT_RCU · line 446
Quoted 6 lines
init/Kconfig +3
-0@@ -441,6 +441,16 @@ config PREEMPT_RCU This option enables preemptible-RCU code that is common between the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. +config RCU_USER_QS + bool "Consider userspace as in RCU extended quiescent state" + depends on HAVE_RCU_USER_QS && SMPDoes this actually depend on SMP, or does it depend on the non-TINY RCU implementation? If the latter, it should depend on that rather than SMP.
Quoted 3 lines
(I assume that the tiny RCU implementation simply doesn't need all this machinery because it doesn't need coordinated quiescence at all? Or does tiny RCU still cause a periodic wakeup on UP?)
It actually does depend on SMP. There has to be at least one CPU taking scheduling-clock interrupts in order to keep time computation accurate, so a de-facto UP system cannot adaptive-dynticks its sole CPU.
Thanx, Paul
Quoted 58 lines
Quoted 57 lines
init/Kconfig +7
-0+ help + This option sets hooks on kernel / userspace boundaries and + puts RCU in extended quiescent state when the CPU runs in + userspace. It means that when a CPU runs in userspace, it is + excluded from the global RCU state machine and thus doesn't + to keep the timer tick on for RCU. + config RCU_FANOUT int "Tree-based hierarchical RCU fanout value" range 2 64 if 64BITkernel/rcutree.c +4
-1@@ -424,6 +424,7 @@ void rcu_idle_enter(void) }EXPORT_SYMBOL_GPL (rcu_idle_enter); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_enter - inform RCU that we are resuming userspace. * @@ -438,7 +439,6 @@ void rcu_user_enter(void) }EXPORT_SYMBOL_GPL (rcu_user_enter); -/** * rcu_user_enter_irq - inform RCU that we are going to resume userspace * after the current irq returns. @@ -459,6 +459,7 @@ void rcu_user_enter_irq(void) rdtp->dynticks_nesting =1 ; local_irq_restore(flags); } +#endif /** * rcu_irq_exit - inform RCU that current CPU is exiting irq towards idle @@ -562,6 +563,7 @@ void rcu_idle_exit(void) }EXPORT_SYMBOL_GPL (rcu_idle_exit); +#ifdef CONFIG_RCU_USER_QS /** * rcu_user_exit - inform RCU that we are exiting userspace. * @@ -595,6 +597,7 @@ void rcu_user_exit_irq(void) rdtp->dynticks_nesting +=DYNTICK_TASK_EXIT_IDLE ; local_irq_restore(flags); } +#endif /** * rcu_irq_enter - inform RCU that current CPU is entering irq away from idle Signature · 1 line
1.7.8
Josh Triplett
maintainerRe: [PATCH tip/core/rcu 04/26] rcu: Settle config for userspace extended quiescent state
Quoted 99 lines · Paul E. McKenney
Quoted 98 lines · Josh Triplett
Quoted 20 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Create a new config option under the RCU menu that put CPUs under RCU extended quiescent state (as in dynticks idle mode) when they run in userspace. This require some contribution from architectures to hook into kernel and userspace boundaries.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>One question below, but nonethelesss:
Reviewed-by Josh Triplett <[email protected]>Quoted 53 lines
arch/Kconfig | 10 ++++++++++ include/linux/rcupdate.h | 8 ++++++++ init/Kconfig | 10 ++++++++++ kernel/rcutree.c | 5 ++++- 4 files changed, 32 insertions(+), 1 deletions(-)arch/Kconfig +10
-0@@ -281,4 +281,14 @@ config SECCOMP_FILTER See Documentation/prctl/seccomp_filter.txt for details. +config HAVE_RCU_USER_QS + bool + help + Provide kernel entry/exit hooks necessary for userspace + RCU extended quiescent state. Syscalls need to be wrapped inside + rcu_user_exit()-rcu_user_enter() through the slow path using + TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs + are already protected inside rcu_irq_enter/rcu_irq_exit() but + preemption or signal handling on irq exit still need to be protected. + source "kernel/gcov/Kconfig"include/linux/rcupdate.h +8
-0@@ -191,10 +191,18 @@ extern void rcu_idle_enter(void);extern void rcu_idle_exit (void );extern void rcu_irq_enter (void );extern void rcu_irq_exit (void ); + +#ifdef CONFIG_RCU_USER_QS extern void rcu_user_enter (void );extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +#else +static inline void rcu_user_enter (void ) { } +static inline void rcu_user_exit (void ) { } +#endif /* CONFIG_RCU_USER_QS */ + +extern void exit_rcu (void );/** init/Kconfig +0
-0init/Kconfig config PREEMPT_RCU · line 446
Quoted 6 lines
init/Kconfig +3
-0@@ -441,6 +441,16 @@ config PREEMPT_RCU This option enables preemptible-RCU code that is common between the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. +config RCU_USER_QS + bool "Consider userspace as in RCU extended quiescent state" + depends on HAVE_RCU_USER_QS && SMPDoes this actually depend on SMP, or does it depend on the non-TINY RCU implementation? If the latter, it should depend on that rather than SMP.
Quoted 7 lines
Quoted 3 lines
(I assume that the tiny RCU implementation simply doesn't need all this machinery because it doesn't need coordinated quiescence at all? Or does tiny RCU still cause a periodic wakeup on UP?)
It actually does depend on SMP. There has to be at least one CPU taking scheduling-clock interrupts in order to keep time computation accurate, so a de-facto UP system cannot adaptive-dynticks its sole CPU.
Ah. That seems like a removable limitation, albeit a difficult one. Nonetheless, it makes sense to avoid providing the option when it won't help.
However, once a config symbol for adaptive dynticks exists, perhaps that symbol should depend on SMP and RCU_USER_QS should depend on that instead, documenting the limitation in the right place and making it easier to find and change eventually.
- Josh Triplett
Re: [PATCH tip/core/rcu 04/26] rcu: Settle config for userspace extended quiescent state
Quoted 4 lines · Josh Triplett
Quoted 3 lines
It actually does depend on SMP. There has to be at least one CPU taking scheduling-clock interrupts in order to keep time computation accurate, so a de-facto UP system cannot adaptive-dynticks its sole CPU.
Quoted 8 lines
Ah. That seems like a removable limitation, albeit a difficult one. Nonetheless, it makes sense to avoid providing the option when it won't help.
However, once a config symbol for adaptive dynticks exists, perhaps that symbol should depend on SMP and
RCU_USER_QSshould depend on that instead, documenting the limitation in the right place and making it easier to find and change eventually.
Right! And in fact CONFIG_RCU_USER_QS is a temporary config. Once we'll have CONFIG_NO_HZ_FULL, we won't need intermediate configs like this. And CONFIG_NO_HZ_FULL will depend on SMP anyway.
PATCH 5/26 rcu: Allow rcu_user_enter()/exit() to nest
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Allow calls to rcu_user_enter() even if we are already in userspace (as seen by RCU) and allow calls to rcu_user_exit() even if we are already in the kernel.
This makes the APIs more flexible to be called from architectures. Exception entries for example won't need to know if they come from userspace before calling rcu_user_exit().
--- kernel/rcutree.c | 41 +++++++++++++++++++++++++++++++++-------- kernel/rcutree.h | 3 +++ 2 files changed, 36 insertions(+), 8 deletions(-)
kernel/rcutree.c +33 -8
@@ -390,11 +390,9 @@ static void rcu_eqs_enter_common(struct rcu_dynticks *rdtp, long long oldval,*/ static void rcu_eqs_enter (bool user) { -unsigned long flags;long long oldval;struct rcu_dynticks *rdtp; -local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); oldval = rdtp->dynticks_nesting;WARN_ON_ONCE ((oldval &DYNTICK_TASK_NEST_MASK ) ==0 );
@@ -403,7 +401,6 @@ static void rcu_eqs_enter(bool user)else rdtp->dynticks_nesting -=DYNTICK_TASK_NEST_VALUE ;rcu_eqs_enter_common (rdtp, oldval, user); -local_irq_restore (flags); }/**
@@ -420,7 +417,11 @@ static void rcu_eqs_enter(bool user)*/ void rcu_idle_enter (void ) { +unsigned long flags; + +local_irq_save (flags);rcu_eqs_enter (0 ); +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_idle_enter);
@@ -435,7 +436,18 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter);*/ void rcu_user_enter (void ) { -rcu_eqs_enter (1 ); +unsigned long flags; +struct rcu_dynticks *rdtp; + +WARN_ON_ONCE (!current->mm); + +local_irq_save (flags); + rdtp = &__get_cpu_var (rcu_dynticks); +if (!rdtp->in_user) { + rdtp->in_user = true; +rcu_eqs_enter (1 ); + } +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_user_enter);
@@ -530,11 +542,9 @@ static void rcu_eqs_exit_common(struct rcu_dynticks *rdtp, long long oldval,*/ static void rcu_eqs_exit (bool user) { -unsigned long flags;struct rcu_dynticks *rdtp;long long oldval; -local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); oldval = rdtp->dynticks_nesting;WARN_ON_ONCE (oldval <0 );
@@ -543,7 +553,6 @@ static void rcu_eqs_exit(bool user)else rdtp->dynticks_nesting =DYNTICK_TASK_EXIT_IDLE ;rcu_eqs_exit_common (rdtp, oldval, user); -local_irq_restore (flags); }/**
@@ -559,7 +568,11 @@ static void rcu_eqs_exit(bool user)*/ void rcu_idle_exit (void ) { +unsigned long flags; + +local_irq_save (flags);rcu_eqs_exit (0 ); +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_idle_exit);
@@ -572,7 +585,16 @@ EXPORT_SYMBOL_GPL(rcu_idle_exit);*/ void rcu_user_exit (void ) { -rcu_eqs_exit (1 ); +unsigned long flags; +struct rcu_dynticks *rdtp; + +local_irq_save (flags); + rdtp = &__get_cpu_var (rcu_dynticks); +if (rdtp->in_user) { + rdtp->in_user = false; +rcu_eqs_exit (1 ); + } +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_user_exit);
@@ -2590,6 +2612,9 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) rdp->dynticks = &per_cpu (rcu_dynticks, cpu);WARN_ON_ONCE (rdp->dynticks->dynticks_nesting !=DYNTICK_TASK_EXIT_IDLE );WARN_ON_ONCE (atomic_read (&rdp->dynticks->dynticks) !=1 ); +#ifdef CONFIG_RCU_USER_QS +WARN_ON_ONCE (rdp->dynticks->in_user); +#endif rdp->cpu = cpu; rdp->rsp = rsp;raw_spin_unlock_irqrestore (&rnp->lock, flags);
kernel/rcutree.h +3 -0
@@ -102,6 +102,9 @@ struct rcu_dynticks {/* idle-period nonlazy_posted snapshot. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ #endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ +#ifdef CONFIG_RCU_USER_QS +bool in_user;/* Is the CPU in userland from RCU POV? */ +#endif };/* RCU's kthread states for tracing. */
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 05/26] rcu: Allow rcu_user_enter()/exit() to nest
Quoted 22 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Allow calls to
rcu_user_enter()even if we are already in userspace (as seen by RCU) and allow calls torcu_user_exit()even if we are already in the kernel.This makes the APIs more flexible to be called from architectures. Exception entries for example won't need to know if they come from userspace before calling
rcu_user_exit().Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 137 lines
kernel/rcutree.c | 41 +++++++++++++++++++++++++++++++++-------- kernel/rcutree.h | 3 +++ 2 files changed, 36 insertions(+), 8 deletions(-)kernel/rcutree.c +33
-8@@ -390,11 +390,9 @@ static void rcu_eqs_enter_common(struct rcu_dynticks *rdtp, long long oldval,*/ static void rcu_eqs_enter (bool user) { -unsigned long flags;long long oldval;struct rcu_dynticks *rdtp; -local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); oldval = rdtp->dynticks_nesting;WARN_ON_ONCE ((oldval &DYNTICK_TASK_NEST_MASK ) ==0 );@@ -403,7 +401,6 @@ static void rcu_eqs_enter(bool user)else rdtp->dynticks_nesting -=DYNTICK_TASK_NEST_VALUE ;rcu_eqs_enter_common (rdtp, oldval, user); -local_irq_restore (flags); }/** @@ -420,7 +417,11 @@ static void rcu_eqs_enter(bool user)*/ void rcu_idle_enter (void ) { +unsigned long flags; + +local_irq_save (flags);rcu_eqs_enter (0 ); +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_idle_enter);@@ -435,7 +436,18 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter);*/ void rcu_user_enter (void ) { -rcu_eqs_enter (1 ); +unsigned long flags; +struct rcu_dynticks *rdtp; + +WARN_ON_ONCE (!current->mm); + +local_irq_save (flags); + rdtp = &__get_cpu_var (rcu_dynticks); +if (!rdtp->in_user) { + rdtp->in_user = true; +rcu_eqs_enter (1 ); + } +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_user_enter);@@ -530,11 +542,9 @@ static void rcu_eqs_exit_common(struct rcu_dynticks *rdtp, long long oldval,*/ static void rcu_eqs_exit (bool user) { -unsigned long flags;struct rcu_dynticks *rdtp;long long oldval; -local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); oldval = rdtp->dynticks_nesting;WARN_ON_ONCE (oldval <0 );@@ -543,7 +553,6 @@ static void rcu_eqs_exit(bool user)else rdtp->dynticks_nesting =DYNTICK_TASK_EXIT_IDLE ;rcu_eqs_exit_common (rdtp, oldval, user); -local_irq_restore (flags); }/** @@ -559,7 +568,11 @@ static void rcu_eqs_exit(bool user)*/ void rcu_idle_exit (void ) { +unsigned long flags; + +local_irq_save (flags);rcu_eqs_exit (0 ); +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_idle_exit);@@ -572,7 +585,16 @@ EXPORT_SYMBOL_GPL(rcu_idle_exit);*/ void rcu_user_exit (void ) { -rcu_eqs_exit (1 ); +unsigned long flags; +struct rcu_dynticks *rdtp; + +local_irq_save (flags); + rdtp = &__get_cpu_var (rcu_dynticks); +if (rdtp->in_user) { + rdtp->in_user = false; +rcu_eqs_exit (1 ); + } +local_irq_restore (flags); }EXPORT_SYMBOL_GPL (rcu_user_exit);@@ -2590,6 +2612,9 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) rdp->dynticks = &per_cpu (rcu_dynticks, cpu);WARN_ON_ONCE (rdp->dynticks->dynticks_nesting !=DYNTICK_TASK_EXIT_IDLE );WARN_ON_ONCE (atomic_read (&rdp->dynticks->dynticks) !=1 ); +#ifdef CONFIG_RCU_USER_QS +WARN_ON_ONCE (rdp->dynticks->in_user); +#endif rdp->cpu = cpu; rdp->rsp = rsp;raw_spin_unlock_irqrestore (&rnp->lock, flags);kernel/rcutree.h +3
-0@@ -102,6 +102,9 @@ struct rcu_dynticks {/* idle-period nonlazy_posted snapshot. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ #endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ +#ifdef CONFIG_RCU_USER_QS +bool in_user;/* Is the CPU in userland from RCU POV? */ +#endif };/* RCU's kthread states for tracing. */ Signature · 1 line
1.7.8
PATCH 6/26 rcu: Ignore userspace extended quiescent state by default
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
By default we don't want to enter into RCU extended quiescent state while in userspace because doing this produces some overhead (eg: use of syscall slowpath). Set it off by default and ready to run when some feature like adaptive tickless need it.
--- kernel/rcutree.c | 5 ++++- kernel/rcutree.h | 1 + 2 files changed, 5 insertions(+), 1 deletions(-)
kernel/rcutree.c +4 -1
@@ -210,6 +210,9 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = { .dynticks_nesting =DYNTICK_TASK_EXIT_IDLE , .dynticks =ATOMIC_INIT (1 ), +#ifdef CONFIG_RCU_USER_QS + .ignore_user_qs = true, +#endif };static int blimit =10 ;/* Maximum callbacks per rcu_do_batch. */
@@ -443,7 +446,7 @@ void rcu_user_enter(void)local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); -if (!rdtp->in_user) { +if (!rdtp->ignore_user_qs && !rdtp->in_user) { rdtp->in_user = true;rcu_eqs_enter (1 ); }
kernel/rcutree.h +1 -0
@@ -103,6 +103,7 @@ struct rcu_dynticks {int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ #endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ #ifdef CONFIG_RCU_USER_QS +bool ignore_user_qs;/* Treat userspace as extended QS or not */ bool in_user;/* Is the CPU in userland from RCU POV? */ #endif };
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 06/26] rcu: Ignore userspace extended quiescent state by default
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
By default we don't want to enter into RCU extended quiescent state while in userspace because doing this produces some overhead (eg: use of syscall slowpath). Set it off by default and ready to run when some feature like adaptive tickless need it.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 42 lines
kernel/rcutree.c | 5 ++++- kernel/rcutree.h | 1 + 2 files changed, 5 insertions(+), 1 deletions(-)kernel/rcutree.c +4
-1@@ -210,6 +210,9 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = { .dynticks_nesting =DYNTICK_TASK_EXIT_IDLE , .dynticks =ATOMIC_INIT (1 ), +#ifdef CONFIG_RCU_USER_QS + .ignore_user_qs = true, +#endif };static int blimit =10 ;/* Maximum callbacks per rcu_do_batch. */ @@ -443,7 +446,7 @@ void rcu_user_enter(void)local_irq_save (flags); rdtp = &__get_cpu_var (rcu_dynticks); -if (!rdtp->in_user) { +if (!rdtp->ignore_user_qs && !rdtp->in_user) { rdtp->in_user = true;rcu_eqs_enter (1 ); }kernel/rcutree.h +1
-0@@ -103,6 +103,7 @@ struct rcu_dynticks {int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */ #endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ #ifdef CONFIG_RCU_USER_QS +bool ignore_user_qs;/* Treat userspace as extended QS or not */ bool in_user;/* Is the CPU in userland from RCU POV? */ #endif };Signature · 1 line
1.7.8
PATCH 7/26 rcu: Switch task's syscall hooks on context switch
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Clear the syscalls hook of a task when it's scheduled out so that if the task migrates, it doesn't run the syscall slow path on a CPU that might not need it.
Also set the syscalls hook on the next task if needed.
--- arch/um/drivers/mconsole_kern.c | 1 + include/linux/rcupdate.h | 2 ++ include/linux/sched.h | 8 ++++++++ kernel/rcutree.c | 15 +++++++++++++++ kernel/sched/core.c | 1 + 5 files changed, 27 insertions(+), 0 deletions(-)
arch/um/drivers/mconsole_kern.c +1 -0
@@ -705,6 +705,7 @@ static void stack_proc(void *arg)struct task_struct *from = current, *to = arg; to->thread.saved_task = from; +rcu_switch (from, to);switch_to (from, to, from); }
include/linux/rcupdate.h +2 -0
@@ -197,6 +197,8 @@ extern void rcu_user_enter(void);extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +extern void rcu_user_hooks_switch (struct task_struct *prev, +struct task_struct *next);#else static inline void rcu_user_enter (void ) { }static inline void rcu_user_exit (void ) { }
include/linux/sched.h +8 -0
@@ -1894,6 +1894,14 @@ static inline void rcu_copy_process(struct task_struct *p)#endif +static inline void rcu_switch (struct task_struct *prev, +struct task_struct *next) +{ +#ifdef CONFIG_RCU_USER_QS +rcu_user_hooks_switch (prev, next); +#endif +} +static inline void tsk_restore_flags (struct task_struct *task,unsigned long orig_flags,unsigned long flags) {
kernel/rcutree.c +15 -0
@@ -721,6 +721,21 @@ int rcu_is_cpu_idle(void) }EXPORT_SYMBOL (rcu_is_cpu_idle); +#ifdef CONFIG_RCU_USER_QS +void rcu_user_hooks_switch (struct task_struct *prev, +struct task_struct *next) +{ +struct rcu_dynticks *rdtp; + +/* Interrupts are disabled in context switch */ + rdtp = &__get_cpu_var (rcu_dynticks); +if (!rdtp->ignore_user_qs) { +clear_tsk_thread_flag (prev,TIF_NOHZ ); +set_tsk_thread_flag (next,TIF_NOHZ ); + } +} +#endif /* #ifdef CONFIG_RCU_USER_QS */ +#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_HOTPLUG_CPU )/*
kernel/sched/core.c +1 -0
@@ -2081,6 +2081,7 @@ context_switch(struct rq *rq, struct task_struct *prev,#endif /* Here we just switch the register state and the stack. */ +rcu_switch (prev, next);switch_to (prev, next, prev);barrier ();
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 07/26] rcu: Switch task's syscall hooks on context switch
Quoted 20 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Clear the syscalls hook of a task when it's scheduled out so that if the task migrates, it doesn't run the syscall slow path on a CPU that might not need it.
Also set the syscalls hook on the next task if needed.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 92 lines
arch/um/drivers/mconsole_kern.c | 1 + include/linux/rcupdate.h | 2 ++ include/linux/sched.h | 8 ++++++++ kernel/rcutree.c | 15 +++++++++++++++ kernel/sched/core.c | 1 + 5 files changed, 27 insertions(+), 0 deletions(-)arch/um/drivers/mconsole_kern.c +1
-0@@ -705,6 +705,7 @@ static void stack_proc(void *arg)struct task_struct *from = current, *to = arg; to->thread.saved_task = from; +rcu_switch (from, to);switch_to (from, to, from); }include/linux/rcupdate.h +2
-0@@ -197,6 +197,8 @@ extern void rcu_user_enter(void);extern void rcu_user_exit (void );extern void rcu_user_enter_irq (void );extern void rcu_user_exit_irq (void ); +extern void rcu_user_hooks_switch (struct task_struct *prev, +struct task_struct *next);#else static inline void rcu_user_enter (void ) { }static inline void rcu_user_exit (void ) { }include/linux/sched.h +8
-0@@ -1894,6 +1894,14 @@ static inline void rcu_copy_process(struct task_struct *p)#endif +static inline void rcu_switch (struct task_struct *prev, +struct task_struct *next) +{ +#ifdef CONFIG_RCU_USER_QS +rcu_user_hooks_switch (prev, next); +#endif +} +static inline void tsk_restore_flags (struct task_struct *task,unsigned long orig_flags,unsigned long flags) {kernel/rcutree.c +15
-0@@ -721,6 +721,21 @@ int rcu_is_cpu_idle(void) }EXPORT_SYMBOL (rcu_is_cpu_idle); +#ifdef CONFIG_RCU_USER_QS +void rcu_user_hooks_switch (struct task_struct *prev, +struct task_struct *next) +{ +struct rcu_dynticks *rdtp; + +/* Interrupts are disabled in context switch */ + rdtp = &__get_cpu_var (rcu_dynticks); +if (!rdtp->ignore_user_qs) { +clear_tsk_thread_flag (prev,TIF_NOHZ ); +set_tsk_thread_flag (next,TIF_NOHZ ); + } +} +#endif /* #ifdef CONFIG_RCU_USER_QS */ +#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_HOTPLUG_CPU )/* kernel/sched/core.c +1
-0@@ -2081,6 +2081,7 @@ context_switch(struct rq *rq, struct task_struct *prev,#endif /* Here we just switch the register state and the stack. */ +rcu_switch (prev, next);switch_to (prev, next, prev);barrier ();Signature · 1 line
1.7.8
PATCH 8/26 x86: Syscall hooks for userspace RCU extended QS
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Add syscall slow path hooks to notify syscall entry and exit on CPUs that want to support userspace RCU extended quiescent state.
--- arch/x86/include/asm/thread_info.h | 10 +++++++--- arch/x86/kernel/ptrace.c | 5 +++++ 2 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/include/asm/thread_info.h +7 -3
@@ -89,6 +89,7 @@ struct thread_info {#define TIF_NOTSC 16/* TSC is not accessible in userland */ #define TIF_IA32 17/* IA32 compatibility process */ #define TIF_FORK 18/* ret_from_fork */ +#define TIF_NOHZ 19/* in adaptive nohz mode */ #define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_DEBUG 21/* uses debug registers */ #define TIF_IO_BITMAP 22/* uses I/O bitmap */
@@ -114,6 +115,7 @@ struct thread_info {#define _TIF_NOTSC (1 << TIF_NOTSC)#define _TIF_IA32 (1 << TIF_IA32)#define _TIF_FORK (1 << TIF_FORK) +#define _TIF_NOHZ (1 << TIF_NOHZ)#define _TIF_DEBUG (1 << TIF_DEBUG)#define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP)#define _TIF_FORCED_TF (1 << TIF_FORCED_TF)
@@ -126,12 +128,13 @@ struct thread_info {/* work to do in syscall_trace_enter() */ #define _TIF_WORK_SYSCALL_ENTRY \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_EMU | _TIF_SYSCALL_AUDIT | \ - _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT) + _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* work to do in syscall_trace_leave() */ #define _TIF_WORK_SYSCALL_EXIT \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SINGLESTEP | \ - _TIF_SYSCALL_TRACEPOINT) + _TIF_SYSCALL_TRACEPOINT | _TIF_NOHZ)/* work to do on interrupt/exception return */ #define _TIF_WORK_MASK \
@@ -141,7 +144,8 @@ struct thread_info {/* work to do on any return to user space */ #define _TIF_ALLWORK_MASK \ - ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT) + ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* Only used for 64 bit */ #define _TIF_DO_NOTIFY_MASK \
arch/x86/kernel/ptrace.c +5 -0
@@ -21,6 +21,7 @@#include <linux/signal.h> #include <linux/perf_event.h> #include <linux/hw_breakpoint.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/pgtable.h>
@@ -1463,6 +1464,8 @@ long syscall_trace_enter(struct pt_regs *regs) {long ret =0 ; +rcu_user_exit (); +/* * If we stepped into a sysenter/syscall insn, it trapped in * kernel mode; do_debug() cleared TF and set TIF_SINGLESTEP.
@@ -1526,4 +1529,6 @@ void syscall_trace_leave(struct pt_regs *regs) !test_thread_flag (TIF_SYSCALL_EMU );if (step ||test_thread_flag (TIF_SYSCALL_TRACE ))tracehook_report_syscall_exit (regs, step); + +rcu_user_enter (); }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 08/26] x86: Syscall hooks for userspace RCU extended QS
Quoted 18 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Add syscall slow path hooks to notify syscall entry and exit on CPUs that want to support userspace RCU extended quiescent state.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
This seems reasonable; presumably you plan to add something actually setting TIF_NOHZ in a subsequent patch series?
Quoted 81 lines
arch/x86/include/asm/thread_info.h | 10 +++++++--- arch/x86/kernel/ptrace.c | 5 +++++ 2 files changed, 12 insertions(+), 3 deletions(-)arch/x86/include/asm/thread_info.h +7
-3@@ -89,6 +89,7 @@ struct thread_info {#define TIF_NOTSC 16/* TSC is not accessible in userland */ #define TIF_IA32 17/* IA32 compatibility process */ #define TIF_FORK 18/* ret_from_fork */ +#define TIF_NOHZ 19/* in adaptive nohz mode */ #define TIF_MEMDIE 20/* is terminating due to OOM killer */ #define TIF_DEBUG 21/* uses debug registers */ #define TIF_IO_BITMAP 22/* uses I/O bitmap */ @@ -114,6 +115,7 @@ struct thread_info {#define _TIF_NOTSC (1 << TIF_NOTSC)#define _TIF_IA32 (1 << TIF_IA32)#define _TIF_FORK (1 << TIF_FORK) +#define _TIF_NOHZ (1 << TIF_NOHZ)#define _TIF_DEBUG (1 << TIF_DEBUG)#define _TIF_IO_BITMAP (1 << TIF_IO_BITMAP)#define _TIF_FORCED_TF (1 << TIF_FORCED_TF)@@ -126,12 +128,13 @@ struct thread_info {/* work to do in syscall_trace_enter() */ #define _TIF_WORK_SYSCALL_ENTRY \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_EMU | _TIF_SYSCALL_AUDIT | \ - _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT) + _TIF_SECCOMP | _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* work to do in syscall_trace_leave() */ #define _TIF_WORK_SYSCALL_EXIT \ (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | _TIF_SINGLESTEP | \ - _TIF_SYSCALL_TRACEPOINT) + _TIF_SYSCALL_TRACEPOINT | _TIF_NOHZ)/* work to do on interrupt/exception return */ #define _TIF_WORK_MASK \@@ -141,7 +144,8 @@ struct thread_info {/* work to do on any return to user space */ #define _TIF_ALLWORK_MASK \ - ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT) + ((0x0000FFFF & ~_TIF_SECCOMP) | _TIF_SYSCALL_TRACEPOINT | \ + _TIF_NOHZ)/* Only used for 64 bit */ #define _TIF_DO_NOTIFY_MASK \arch/x86/kernel/ptrace.c +5
-0@@ -21,6 +21,7 @@#include <linux/signal.h> #include <linux/perf_event.h> #include <linux/hw_breakpoint.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/pgtable.h> @@ -1463,6 +1464,8 @@ long syscall_trace_enter(struct pt_regs *regs) {long ret =0 ; +rcu_user_exit (); +/* * If we stepped into a sysenter/syscall insn, it trapped in * kernel mode; do_debug() cleared TF and set TIF_SINGLESTEP. @@ -1526,4 +1529,6 @@ void syscall_trace_leave(struct pt_regs *regs) !test_thread_flag (TIF_SYSCALL_EMU );if (step ||test_thread_flag (TIF_SYSCALL_TRACE ))tracehook_report_syscall_exit (regs, step); + +rcu_user_enter (); }Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 08/26] x86: Syscall hooks for userspace RCU extended QS
Quoted 28 lines · Josh Triplett
Quoted 18 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Add syscall slow path hooks to notify syscall entry and exit on CPUs that want to support userspace RCU extended quiescent state.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 2 lines
This seems reasonable; presumably you plan to add something actually setting
TIF_NOHZin a subsequent patch series?
Yes, currently this is set by the selftests, see the combination of:
[PATCH tip/core/rcu 07/26] rcu: Switch task's syscall hooks on context switch [PATCH tip/core/rcu 14/26] rcu: Userspace RCU extended QS selftest
Later this will be driven by the nohz subsystem.
PATCH 9/26 x86: Exception hooks for userspace RCU extended QS
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Add necessary hooks to x86 exception for userspace RCU extended quiescent state support.
This includes traps, page fault, debug exceptions, etc...
--- arch/x86/include/asm/rcu.h | 20 ++++++++++++++++++++ arch/x86/kernel/traps.c | 30 ++++++++++++++++++++++-------- arch/x86/mm/fault.c | 13 +++++++++++-- 3 files changed, 53 insertions(+), 10 deletions(-) create mode 100644 arch/x86/include/asm/rcu.h
arch/x86/include/asm/rcu.h new file +20 -0
@@ -0,0 +1,20 @@ +#ifndef _ASM_X86_RCU_H +#define _ASM_X86_RCU_H + +#include <linux/rcupdate.h> +#include <asm/ptrace.h> + +static inline void exception_enter (struct pt_regs *regs) +{ +rcu_user_exit (); +} + +static inline void exception_exit (struct pt_regs *regs) +{ +#ifdef CONFIG_RCU_USER_QS +if (user_mode (regs)) +rcu_user_enter (); +#endif +} + +#endif
arch/x86/kernel/traps.c +22 -8
@@ -55,6 +55,7 @@#include <asm/i387.h> #include <asm/fpu-internal.h> #include <asm/mce.h> +#include <asm/rcu.h> #include <asm/mach_traps.h>
@@ -180,11 +181,15 @@ vm86_trap:#define DO_ERROR (trapnr, signr, str, name) \ dotraplinkage void do_##name(struct pt_regs *regs, long error_code) \ { \ - if (notify_die(DIE_TRAP, str, regs, error_code, trapnr, signr) \ - == NOTIFY_STOP) \ + exception_enter(regs); \ + if (notify_die(DIE_TRAP, str, regs, error_code, \ + trapnr, signr) == NOTIFY_STOP) { \ + exception_exit(regs); \ return; \ + } \ conditional_sti(regs); \ do_trap(trapnr, signr, str, regs, error_code, NULL); \ + exception_exit(regs); \ }#define DO_ERROR_INFO (trapnr, signr, str, name, sicode, siaddr) \
@@ -195,11 +200,15 @@ dotraplinkage void do_##name(struct pt_regs *regs, long error_code) \ info.si_errno =0 ; \ info.si_code = sicode; \ info.si_addr = (void __user *)siaddr; \ -if (notify_die (DIE_TRAP , str, regs, error_code, trapnr, signr) \ - ==NOTIFY_STOP ) \ +exception_enter (regs); \ +if (notify_die (DIE_TRAP , str, regs, error_code, \ + trapnr, signr) ==NOTIFY_STOP ) { \ +exception_exit (regs); \return ; \ + } \conditional_sti (regs); \do_trap (trapnr, signr, str, regs, error_code, &info); \ +exception_exit (regs); \ } DO_ERROR_INFO(X86_TRAP_DE, SIGFPE, "divide error", divide_error, FPE_INTDIV,
@@ -312,6 +321,7 @@ dotraplinkage void __kprobes notrace do_int3(struct pt_regs *regs, long error_co ftrace_int3_handler(regs))return ;#endif +exception_enter (regs);#ifdef CONFIG_KGDB_LOW_LEVEL_TRAP if (kgdb_ll_trap (DIE_INT3 ,"int3" , regs, error_code,X86_TRAP_BP ,SIGTRAP ) ==NOTIFY_STOP )
@@ -331,6 +341,7 @@ dotraplinkage void __kprobes notrace do_int3(struct pt_regs *regs, long error_codo_trap (X86_TRAP_BP ,SIGTRAP ,"int3" , regs, error_code,NULL );preempt_conditional_cli (regs);debug_stack_usage_dec (); +exception_exit (regs); }#ifdef CONFIG_X86_64
@@ -391,6 +402,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)unsigned long dr6;int si_code; +exception_enter (regs); +get_debugreg (dr6,6 );/* Filter out all the reserved bits which are preset to 1 */
@@ -406,7 +419,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)/* Catch kmemcheck conditions first of all! */ if ((dr6 &DR_STEP ) &&kmemcheck_trap (regs)) -return ; + goto exit;/* DR6 may or may not be cleared by the CPU */ set_debugreg (0 ,6 );
@@ -421,7 +434,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)if (notify_die (DIE_DEBUG ,"debug" , regs,PTR_ERR (&dr6), error_code,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;/* * Let others (NMI) know that the debug stack is in use
@@ -437,7 +450,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code) X86_TRAP_DB);preempt_conditional_cli (regs);debug_stack_usage_dec (); -return ; + goto exit; }/*
@@ -458,7 +471,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)preempt_conditional_cli (regs);debug_stack_usage_dec (); -return ; +exit: +exception_exit (regs); }/*
arch/x86/mm/fault.c +11 -2
@@ -18,6 +18,7 @@#include <asm/pgalloc.h> /* pgd_*(), ... */ #include <asm/kmemcheck.h> /* kmemcheck_*(), ... */ #include <asm/fixmap.h> /* VSYSCALL_START */ +#include <asm/rcu.h> /* exception_enter(), ... */ /* * Page fault error code bits:
@@ -1000,8 +1001,8 @@ static int fault_in_kernel_space(unsigned long address)* and the problem, and then passes it off to one of the appropriate * routines. */ -dotraplinkage void __kprobes -do_page_fault (struct pt_regs *regs,unsigned long error_code) +static void __kprobes +__do_page_fault (struct pt_regs *regs,unsigned long error_code) {struct vm_area_struct *vma;struct task_struct *tsk;
@@ -1209,3 +1210,11 @@ good_area:up_read (&mm->mmap_sem); } + +dotraplinkage void __kprobes +do_page_fault (struct pt_regs *regs,unsigned long error_code) +{ +exception_enter (regs); +__do_page_fault (regs, error_code); +exception_exit (regs); +}
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 09/26] x86: Exception hooks for userspace RCU extended QS
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Add necessary hooks to x86 exception for userspace RCU extended quiescent state support.
This includes traps, page fault, debug exceptions, etc...
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 180 lines
arch/x86/include/asm/rcu.h | 20 ++++++++++++++++++++ arch/x86/kernel/traps.c | 30 ++++++++++++++++++++++-------- arch/x86/mm/fault.c | 13 +++++++++++-- 3 files changed, 53 insertions(+), 10 deletions(-) create mode 100644 arch/x86/include/asm/rcu.harch/x86/include/asm/rcu.h new file +20
-0@@ -0,0 +1,20 @@ +#ifndef _ASM_X86_RCU_H +#define _ASM_X86_RCU_H + +#include <linux/rcupdate.h> +#include <asm/ptrace.h> + +static inline void exception_enter (struct pt_regs *regs) +{ +rcu_user_exit (); +} + +static inline void exception_exit (struct pt_regs *regs) +{ +#ifdef CONFIG_RCU_USER_QS +if (user_mode (regs)) +rcu_user_enter (); +#endif +} + +#endif arch/x86/kernel/traps.c +22
-8@@ -55,6 +55,7 @@#include <asm/i387.h> #include <asm/fpu-internal.h> #include <asm/mce.h> +#include <asm/rcu.h> #include <asm/mach_traps.h> @@ -180,11 +181,15 @@ vm86_trap:#define DO_ERROR (trapnr, signr, str, name) \ dotraplinkage void do_##name(struct pt_regs *regs, long error_code) \ { \ - if (notify_die(DIE_TRAP, str, regs, error_code, trapnr, signr) \ - == NOTIFY_STOP) \ + exception_enter(regs); \ + if (notify_die(DIE_TRAP, str, regs, error_code, \ + trapnr, signr) == NOTIFY_STOP) { \ + exception_exit(regs); \ return; \ + } \ conditional_sti(regs); \ do_trap(trapnr, signr, str, regs, error_code, NULL); \ + exception_exit(regs); \ }#define DO_ERROR_INFO (trapnr, signr, str, name, sicode, siaddr) \@@ -195,11 +200,15 @@ dotraplinkage void do_##name(struct pt_regs *regs, long error_code) \ info.si_errno =0 ; \ info.si_code = sicode; \ info.si_addr = (void __user *)siaddr; \ -if (notify_die (DIE_TRAP , str, regs, error_code, trapnr, signr) \ - ==NOTIFY_STOP ) \ +exception_enter (regs); \ +if (notify_die (DIE_TRAP , str, regs, error_code, \ + trapnr, signr) ==NOTIFY_STOP ) { \ +exception_exit (regs); \return ; \ + } \conditional_sti (regs); \do_trap (trapnr, signr, str, regs, error_code, &info); \ +exception_exit (regs); \ } DO_ERROR_INFO(X86_TRAP_DE, SIGFPE, "divide error", divide_error, FPE_INTDIV,@@ -312,6 +321,7 @@ dotraplinkage void __kprobes notrace do_int3(struct pt_regs *regs, long error_co ftrace_int3_handler(regs))return ;#endif +exception_enter (regs);#ifdef CONFIG_KGDB_LOW_LEVEL_TRAP if (kgdb_ll_trap (DIE_INT3 ,"int3" , regs, error_code,X86_TRAP_BP ,SIGTRAP ) ==NOTIFY_STOP )@@ -331,6 +341,7 @@ dotraplinkage void __kprobes notrace do_int3(struct pt_regs *regs, long error_codo_trap (X86_TRAP_BP ,SIGTRAP ,"int3" , regs, error_code,NULL );preempt_conditional_cli (regs);debug_stack_usage_dec (); +exception_exit (regs); }#ifdef CONFIG_X86_64 @@ -391,6 +402,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)unsigned long dr6;int si_code; +exception_enter (regs); +get_debugreg (dr6,6 );/* Filter out all the reserved bits which are preset to 1 */ @@ -406,7 +419,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)/* Catch kmemcheck conditions first of all! */ if ((dr6 &DR_STEP ) &&kmemcheck_trap (regs)) -return ; + goto exit;/* DR6 may or may not be cleared by the CPU */ set_debugreg (0 ,6 );@@ -421,7 +434,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)if (notify_die (DIE_DEBUG ,"debug" , regs,PTR_ERR (&dr6), error_code,SIGTRAP ) ==NOTIFY_STOP ) -return ; + goto exit;/* * Let others (NMI) know that the debug stack is in use @@ -437,7 +450,7 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code) X86_TRAP_DB);preempt_conditional_cli (regs);debug_stack_usage_dec (); -return ; + goto exit; }/* @@ -458,7 +471,8 @@ dotraplinkage void __kprobes do_debug(struct pt_regs *regs, long error_code)preempt_conditional_cli (regs);debug_stack_usage_dec (); -return ; +exit: +exception_exit (regs); }/* arch/x86/mm/fault.c +11
-2@@ -18,6 +18,7 @@#include <asm/pgalloc.h> /* pgd_*(), ... */ #include <asm/kmemcheck.h> /* kmemcheck_*(), ... */ #include <asm/fixmap.h> /* VSYSCALL_START */ +#include <asm/rcu.h> /* exception_enter(), ... */ /* * Page fault error code bits: @@ -1000,8 +1001,8 @@ static int fault_in_kernel_space(unsigned long address)* and the problem, and then passes it off to one of the appropriate * routines. */ -dotraplinkage void __kprobes -do_page_fault (struct pt_regs *regs,unsigned long error_code) +static void __kprobes +__do_page_fault (struct pt_regs *regs,unsigned long error_code) {struct vm_area_struct *vma;struct task_struct *tsk;@@ -1209,3 +1210,11 @@ good_area:up_read (&mm->mmap_sem); } + +dotraplinkage void __kprobes +do_page_fault (struct pt_regs *regs,unsigned long error_code) +{ +exception_enter (regs); +__do_page_fault (regs, error_code); +exception_exit (regs); +}Signature · 1 line
1.7.8
PATCH 10/26 rcu: Exit RCU extended QS on kernel preemption after irq/exception
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
When an exception or an irq exits, and we are going to resume into interrupted kernel code, the low level architecture code calls preempt_schedule_irq() if there is a need to reschedule.
If the interrupt/exception occured between a call to rcu_user_enter() (from syscall exit, exception exit, do_notify_resume exit, ...) and a real resume to userspace (iret,...), preempt_schedule_irq() can be called whereas RCU thinks we are in userspace. But preempt_schedule_irq() is going to run kernel code and may be some RCU read side critical section. We must exit the userspace extended quiescent state before we call it.
To solve this, just call rcu_user_exit() in the beginning of preempt_schedule_irq().
--- kernel/sched/core.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/sched/core.c +1 -0
@@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)/* Catch callers which need to be fixed */ BUG_ON (ti->preempt_count || !irqs_disabled ()); +rcu_user_exit (); do { add_preempt_count(PREEMPT_ACTIVE); local_irq_enable();
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 10/26] rcu: Exit RCU extended QS on kernel preemption after irq/exception
Quoted 29 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When an exception or an irq exits, and we are going to resume into interrupted kernel code, the low level architecture code calls
preempt_schedule_irq()if there is a need to reschedule.If the interrupt/exception occured between a call to
rcu_user_enter()(from syscall exit, exception exit,do_notify_resumeexit, ...) and a real resume to userspace (iret,...),preempt_schedule_irq()can be called whereas RCU thinks we are in userspace. Butpreempt_schedule_irq()is going to run kernel code and may be some RCU read side critical section. We must exit the userspace extended quiescent state before we call it.To solve this, just call
rcu_user_exit()in the beginning ofpreempt_schedule_irq().Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 18 lines
kernel/sched/core.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)kernel/sched/core.c +1
-0@@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)/* Catch callers which need to be fixed */ BUG_ON (ti->preempt_count || !irqs_disabled ()); +rcu_user_exit (); do { add_preempt_count(PREEMPT_ACTIVE); local_irq_enable();Signature · 1 line
1.7.8
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 10/26] rcu: Exit RCU extended QS on kernel preemption after irq/exception
In reply to Paul E. McKenneyQuoted 26 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When an exception or an irq exits, and we are going to resume into interrupted kernel code, the low level architecture code calls
preempt_schedule_irq()if there is a need to reschedule.If the interrupt/exception occured between a call to
rcu_user_enter()(from syscall exit, exception exit,do_notify_resumeexit, ...) and a real resume to userspace (iret,...),preempt_schedule_irq()can be called whereas RCU thinks we are in userspace. Butpreempt_schedule_irq()is going to run kernel code and may be some RCU read side critical section. We must exit the userspace extended quiescent state before we call it.To solve this, just call
rcu_user_exit()in the beginning ofpreempt_schedule_irq().--- kernel/sched/core.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c asmlinkage void __sched preempt_schedule_irq(void) · line 3570
Quoted 7 lines
kernel/sched/core.c +1
-0@@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)/* Catch callers which need to be fixed */ BUG_ON (ti->preempt_count || !irqs_disabled ()); +rcu_user_exit (); do { add_preempt_count(PREEMPT_ACTIVE); local_irq_enable();
But in this case you want to restart the tick as well, are we going to add a nohz_user_exit() like call as well, or do we have something that does both?
Re: [PATCH tip/core/rcu 10/26] rcu: Exit RCU extended QS on kernel preemption after irq/exception
Quoted 30 lines · Peter Zijlstra
Quoted 29 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When an exception or an irq exits, and we are going to resume into interrupted kernel code, the low level architecture code calls
preempt_schedule_irq()if there is a need to reschedule.If the interrupt/exception occured between a call to
rcu_user_enter()(from syscall exit, exception exit,do_notify_resumeexit, ...) and a real resume to userspace (iret,...),preempt_schedule_irq()can be called whereas RCU thinks we are in userspace. Butpreempt_schedule_irq()is going to run kernel code and may be some RCU read side critical section. We must exit the userspace extended quiescent state before we call it.To solve this, just call
rcu_user_exit()in the beginning ofpreempt_schedule_irq().--- kernel/sched/core.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)kernel/sched/core.c +0
-0@@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)/* Catch callers which need to be fixed */ BUG_ON (ti->preempt_count || !irqs_disabled ());
Quoted 8 lines
kernel/sched/core.c asmlinkage void __sched preempt_schedule_irq(void) · line 3570
Quoted 4 lines
kernel/sched/core.c +1
-0+rcu_user_exit (); do { add_preempt_count(PREEMPT_ACTIVE); local_irq_enable();But in this case you want to restart the tick as well, are we going to add a
nohz_user_exit()like call as well, or do we have something that does both?
Yeah this will soon become user_exit() and that hook will rely on the nohz engine and RCU APIs accordingly.
For now I'm only focusing on the RCU user mode. In this patchset it does the RCU user mode unconditionally. I'm just doing this to integrate the things incrementally.
First RCU, then we turn that into some generic user hooks, then we handle the cputime with a generic vtime, then we integrate the nohz engine which drives the user_hooks such that we stop/restart the tick and RCU user mode accordingly.
PATCH 11/26 rcu: Exit RCU extended QS on user preemption
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls schedule() directly.
At that time we may be in extended quiescent state from RCU
POV: the exception is not anymore protected inside rcu_user_exit() - rcu_user_enter() and the irq has called rcu_irq_exit() already.
Create a new API schedule_user() that calls schedule() inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.
--- kernel/sched/core.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/sched/core.c +7 -0
@@ -3463,6 +3463,13 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +asmlinkage void __sched schedule_user(void) +{ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +} +/** * schedule_preempt_disabled - called with preemption disabled *
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 27 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls
schedule()directly.At that time we may be in extended quiescent state from RCU
POV: the exception is not anymore protected insidercu_user_exit()-rcu_user_enter()and the irq has calledrcu_irq_exit()already.Create a new API
schedule_user()that callsschedule()inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 24 lines
kernel/sched/core.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)kernel/sched/core.c +7
-0@@ -3463,6 +3463,13 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +asmlinkage void __sched schedule_user(void) +{ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +} +/** * schedule_preempt_disabled - called with preemption disabled * Signature · 1 line
1.7.8
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
In reply to Paul E. McKenneyQuoted 24 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls
schedule()directly.At that time we may be in extended quiescent state from RCU
POV: the exception is not anymore protected insidercu_user_exit()-rcu_user_enter()and the irq has calledrcu_irq_exit()already.Create a new API
schedule_user()that callsschedule()inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.--- kernel/sched/core.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)kernel/sched/core.c +0
-0
kernel/sched/core.c asmlinkage void __sched schedule(void) · line 3471
Quoted 9 lines
kernel/sched/core.c +6
-0@@ -3463,6 +3463,13 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +asmlinkage void __sched schedule_user(void) +{ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +}
OK, so colour me unconvinced.. why are we doing this?
Typically when we call schedule nr_running != 1 (we need current to be running and a possible target to switch to).
So I'd prefer to simply have schedule() disable all this adaptive tick nonsense and leave it at that.
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 38 lines · Peter Zijlstra
Quoted 24 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls
schedule()directly.At that time we may be in extended quiescent state from RCU
POV: the exception is not anymore protected insidercu_user_exit()-rcu_user_enter()and the irq has calledrcu_irq_exit()already.Create a new API
schedule_user()that callsschedule()inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.--- kernel/sched/core.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)kernel/sched/core.c +0
-0kernel/sched/core.c asmlinkage void __sched schedule(void) · line 3471
Quoted 9 lines
kernel/sched/core.c +6
-0@@ -3463,6 +3463,13 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +asmlinkage void __sched schedule_user(void) +{ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +}OK, so colour me unconvinced.. why are we doing this?
Quoted 5 lines
Typically when we call schedule
nr_running!= 1 (we need current to be running and a possible target to switch to).So I'd prefer to simply have
schedule()disable all this adaptive tick nonsense and leave it at that.
In fact, the only way to get here is through ttwu(), which would have done the nr_running increment and should have disabled all this adaptive stuff.
So again,.. why?
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 42 lines · Peter Zijlstra
Quoted 41 lines · Peter Zijlstra
Quoted 24 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
When exceptions or irq are about to resume userspace, if the task needs to be rescheduled, the arch low level code calls
schedule()directly.At that time we may be in extended quiescent state from RCU
POV: the exception is not anymore protected insidercu_user_exit()-rcu_user_enter()and the irq has calledrcu_irq_exit()already.Create a new API
schedule_user()that callsschedule()inside rcu_user_exit()-rcu_user_enter() in order to protect it. Archs will need to rely on it now to implement user preemption safely.--- kernel/sched/core.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)kernel/sched/core.c +0
-0kernel/sched/core.c asmlinkage void __sched schedule(void) · line 3471
Quoted 9 lines
kernel/sched/core.c +6
-0@@ -3463,6 +3463,13 @@ asmlinkage void __sched schedule(void) }EXPORT_SYMBOL (schedule); +asmlinkage void __sched schedule_user(void) +{ +rcu_user_exit (); +schedule (); +rcu_user_enter (); +}OK, so colour me unconvinced.. why are we doing this?
Typically when we call schedule
nr_running!= 1 (we need current to be running and a possible target to switch to).
Quoted 8 lines
Quoted 2 lines
So I'd prefer to simply have
schedule()disable all this adaptive tick nonsense and leave it at that.In fact, the only way to get here is through
ttwu(), which would have done thenr_runningincrement and should have disabled all this adaptive stuff.So again,.. why?
Ok, indeed if the ttwu happened locally or even remotely through an IPI, the tick engine would stop the tick and exit that RCU extended quiescent state before we reach that place.
I just want to be sure I'm covering every case. There are some places around that call set_need_resched() even if no task is waiting for the CPU (RCU is such an example). In this case the nohz engine won't exit the RCU user mode before schedule().
Another example: a CPU does a remote wake up so it does set_need_resched() and sends the IPI. The target CPU could see the TIF_RESCHED before resuming userspace and call schedule_user() -> schedule() while the IPI has not yet arrived. In this case we need that rcu_user_exit() before schedule might make any use of RCU.
Also when the task returns from schedule(), if it's alone in the runqueue, the rcu_user_enter() that follows can be helpful to stop the tick and enter our RCU quiescent state.
Also I'm considering this RCU user extended quiescent state as a standalone feature for now. Indeed the only user of it is the adaptive tickless thing. But I'm treating that RCU part independantly for now so that it makes it easier to merge, piecewise.
When the adaptive tickless stuff arrives, these rcu_user...() APIs will be replaced by some user_enter() / user_exit() hooks that will all be driven by the nohz subsystem. When we get there the rcu_user_enter() rcu_user_exit() won't be unconditional anymore.
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 29 lines · Frederic Weisbecker
Quoted 5 lines
Quoted 5 lines
OK, so colour me unconvinced.. why are we doing this?
Typically when we call schedule
nr_running!= 1 (we need current to be running and a possible target to switch to).Quoted 8 lines
Quoted 2 lines
So I'd prefer to simply have
schedule()disable all this adaptive tick nonsense and leave it at that.In fact, the only way to get here is through
ttwu(), which would have done thenr_runningincrement and should have disabled all this adaptive stuff.So again,.. why?
Ok, indeed if the ttwu happened locally or even remotely through an IPI, the tick engine would stop the tick and exit that RCU extended quiescent state before we reach that place.
I just want to be sure I'm covering every case. There are some places around that call
set_need_resched()even if no task is waiting for the CPU (RCU is such an example). In this case the nohz engine won't exit the RCU user mode beforeschedule().Another example: a CPU does a remote wake up so it does
set_need_resched()and sends the IPI. The target CPU could see theTIF_RESCHEDbefore resuming userspace and callschedule_user()->schedule()while the IPI has not yet arrived. In this case we need thatrcu_user_exit()before schedule might make any use of RCU.
Quoted 3 lines
Also when the task returns from
schedule(), if it's alone in the runqueue, thercu_user_enter()that follows can be helpful to stop the tick and enter our RCU quiescent state.
So all that should have been in the Changelog.
So the only real problem I see is non IPI remote wakeups, those cannot actually restart the tick on the target cpu like we want.
The rest could all be fixed to not need this.
Hrmm. still don't like the patch, but at least I see why its needed.
Quoted 4 lines
Also I'm considering this RCU user extended quiescent state as a standalone feature for now. Indeed the only user of it is the adaptive tickless thing. But I'm treating that RCU part independantly for now so that it makes it easier to merge, piecewise.
But its semantics are very tightly tied to the requirements of adaptive tick, so arguing about one without the other doesn't really make sense.
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 31 lines · Peter Zijlstra
Quoted 29 lines · Frederic Weisbecker
Quoted 5 lines
Quoted 5 lines
OK, so colour me unconvinced.. why are we doing this?
Typically when we call schedule
nr_running!= 1 (we need current to be running and a possible target to switch to).Quoted 8 lines
Quoted 2 lines
So I'd prefer to simply have
schedule()disable all this adaptive tick nonsense and leave it at that.In fact, the only way to get here is through
ttwu(), which would have done thenr_runningincrement and should have disabled all this adaptive stuff.So again,.. why?
Ok, indeed if the ttwu happened locally or even remotely through an IPI, the tick engine would stop the tick and exit that RCU extended quiescent state before we reach that place.
I just want to be sure I'm covering every case. There are some places around that call
set_need_resched()even if no task is waiting for the CPU (RCU is such an example). In this case the nohz engine won't exit the RCU user mode beforeschedule().Another example: a CPU does a remote wake up so it does
set_need_resched()and sends the IPI. The target CPU could see theTIF_RESCHEDbefore resuming userspace and callschedule_user()->schedule()while the IPI has not yet arrived. In this case we need thatrcu_user_exit()before schedule might make any use of RCU.
Quoted 5 lines
Quoted 3 lines
Also when the task returns from
schedule(), if it's alone in the runqueue, thercu_user_enter()that follows can be helpful to stop the tick and enter our RCU quiescent state.So all that should have been in the Changelog.
Right, I'm fixing that.
Quoted 3 lines
So the only real problem I see is non IPI remote wakeups, those cannot actually restart the tick on the target cpu like we want.
Ok so let identify and validate the corner cases so that I can report them confidently in the changelog:
1) This can happen if something calls set_need_resched() while no other task is on the runqueue.
2) Remote wake up done but we haven't yet received the schedule IPI.
3) Non IPI remote wakeup you're referring above, I'm not sure what you mean though.
Quoted 5 lines
The rest could all be fixed to not need this.
Hrmm. still don't like the patch, but at least I see why its needed.
Quoted 8 lines
Quoted 4 lines
Also I'm considering this RCU user extended quiescent state as a standalone feature for now. Indeed the only user of it is the adaptive tickless thing. But I'm treating that RCU part independantly for now so that it makes it easier to merge, piecewise.
But its semantics are very tightly tied to the requirements of adaptive tick, so arguing about one without the other doesn't really make sense.
Yeah it's true. Now all of these hooks have been required in practice with the adaptive tick code running.
Also keeping these hooks unconditional, until the adaptive tick code comes in and drives that conditionally, is also a nice way to runtime selftest that code (check there is no missing rcu_user_exit() call for example, easy to check if RCU lockdep is enabled).
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 3 lines · Frederic Weisbecker
1) This can happen if something calls
set_need_resched()while no other task is on the runqueue.
People really shouldn't be doing that... I think I know why RCU does this, but yuck. I also think RCU can avoid doing this, but its a toss up if that's worth the trouble.
Quoted 4 lines
2) Remote wake up done but we haven't yet received the schedule IPI.
3) Non IPI remote wakeup you're referring above, I'm not sure what you mean though.
Well there's two ways of doing remote wakeups, one is doing the wakeup from the waking cpu and sending an IPI over to reschedule iff you need wakeup-preemption, the other is queueing the task remotely and sending an IPI to do the wakeup on the remote cpu.
The former has the problem, the latter not.
See ttwu_queue().
We could of course mandate that all remote wakeups to special nohz cpus get queued. That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
So at that point all return to user schedule() calls have nr_running > 1 and the tick is running and RCU is not in extended quiescent state. Since either we had nr_running > 1 and pre and post state are the same, or we had nr_running == 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.
We cannot hit return to user schedule() with nr_running == 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)
Hmm ?
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 11 lines · Peter Zijlstra
We could of course mandate that all remote wakeups to special nohz cpus get queued. That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
So at that point all return to user
schedule()calls havenr_running> 1 and the tick is running and RCU is not in extended quiescent state. Since either we hadnr_running> 1 and pre and post state are the same, or we hadnr_running== 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.
Quoted 5 lines
We cannot hit return to user
schedule()withnr_running== 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)Hmm ?
Crap.. this will screw over -rt, since the wakeups batch the IPI can take forever so we had to disable this.
Bugger it.. I so detest this patch.
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Peter Zijlstra
We could of course mandate that all remote wakeups to special nohz cpus get queued. That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
So at that point all return to user
schedule()calls havenr_running> 1 and the tick is running and RCU is not in extended quiescent state. Since either we hadnr_running> 1 and pre and post state are the same, or we hadnr_running== 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.
Quoted 8 lines
Quoted 5 lines
We cannot hit return to user
schedule()withnr_running== 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)Hmm ?
Crap.. this will screw over -rt, since the wakeups batch the IPI can take forever so we had to disable this.
I don't know that part of -rt. Probably we can deal with that later once we have some upstream code in place?
Quoted 2 lines
Bugger it.. I so detest this patch.
Peter Zijlstra
maintainerRe: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
Quoted 18 lines · Frederic Weisbecker
Quoted 17 lines · Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
We could of course mandate that all remote wakeups to special nohz cpus get queued. That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
So at that point all return to user
schedule()calls havenr_running> 1 and the tick is running and RCU is not in extended quiescent state. Since either we hadnr_running> 1 and pre and post state are the same, or we hadnr_running== 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.We cannot hit return to user
schedule()withnr_running== 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Hmm ?
Crap.. this will screw over -rt, since the wakeups batch the IPI can take forever so we had to disable this.
I don't know that part of -rt. Probably we can deal with that later once we have some upstream code in place?
Yeah maybe.. its going to be hard though, anyway, I'm going to stare at wtf drm is doing for a bit :-)
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
1) This can happen if something calls
set_need_resched()while no other task is on the runqueue.People really shouldn't be doing that... I think I know why RCU does this, but yuck. I also think RCU can avoid doing this, but its a toss up if that's worth the trouble.
There are other places that do this. Look at:
$ git grep set_need_resched drivers
drivers/gpu/drm/i915/i915_gem.c: set_need_resched(); drivers/gpu/drm/ttm/ttm_bo_vm.c: set_need_resched(); drivers/gpu/drm/ttm/ttm_bo_vm.c: set_need_resched(); drivers/gpu/drm/udl/udl_gem.c: set_need_resched();
Quoted 6 lines
Quoted 5 lines
2) Remote wake up done but we haven't yet received the schedule IPI.
3) Non IPI remote wakeup you're referring above, I'm not sure what you mean though.
Quoted 6 lines
Well there's two ways of doing remote wakeups, one is doing the wakeup from the waking cpu and sending an IPI over to reschedule iff you need wakeup-preemption, the other is queueing the task remotely and sending an IPI to do the wakeup on the remote cpu.
The former has the problem, the latter not.
In the former case, if we don't need preemption, we don't call resched_task() and TIF_RESCHED is not set. So the arch code simply doesn't call schedule_user().
If we need wakeup-preemption, then the problem becomes the 2) above.
Am I missing something?
Quoted 5 lines
See
ttwu_queue().We could of course mandate that all remote wakeups to special nohz cpus get queued.
In any case, I think this is good idea to force remote wake ups in nohz cpus, at least when rq->nr_running becomes 2.
In my draft branch, I send an IPI from inc_nr_running() when nr_running becomes 2, so this covers every rq enqueuing scenario, not only wake up. But if we force queued wakeups, I can avoid sending that specific IPI in wake up cases.
Quoted 2 lines
That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
RCU doesn't send anymore IPIs to kick out CPUs that are delaying grace periods. That was not really useful because it wasn't calling set_need_resched(task_cur(cpu)) before doing that. And if it was doing that, we would have missed some bugs
by runtime fixing culprits of stalls.
But RCU calls set_need_resched() from other places like rcu_pending() that is called from rcu_check_callbacks(). IIRC, this is called from the tick.
So if RCU sets TIF_RESCHED from the tick, we may call schedule_user() before that tick resumes userspace.
The other one is on stall detection.
Quoted 6 lines
So at that point all return to user
schedule()calls havenr_running> 1 and the tick is running and RCU is not in extended quiescent state. Since either we hadnr_running> 1 and pre and post state are the same, or we hadnr_running== 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.
If we can guarantee that, but we have yet to make it clear with set_need_resched() callers, then we can certainly remove the rcu_user_exit() call in that function. RCU lockdep would detect what we forgot to think about anyway.
But the rcu_user_enter() call is still valid in the end of schedule_user(). It's going to be useful only if we stop the tick right after returning from that schedule call though. The possible window is very thin so it's probably not that worth the optimization. We can still wait for another tick to stop the timer.
Quoted 4 lines
We cannot hit return to user
schedule()withnr_running== 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)
Sure :)
Quoted 1 line
Hmm ?
Also I forgot one thing: if CONFIG_RCU_USER_QS is not set, I need to call schedule() directly instead of schedule_user(). We don't need that intermediate call in this configuration.
Re: [PATCH tip/core/rcu 11/26] rcu: Exit RCU extended QS on user preemption
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
Quoted 3 lines · Frederic Weisbecker
1) This can happen if something calls
set_need_resched()while no other task is on the runqueue.People really shouldn't be doing that... I think I know why RCU does this, but yuck. I also think RCU can avoid doing this, but its a toss up if that's worth the trouble.
It used to do this whenever it had to force quiescent states on a given CPU more than one time in a given grace period. I have removed this (on your advice), and the only remaining use is during RCU CPU stall warnings, where a given CPU has refused to pass through a quiescent state for more than a minute.
I could remove that one as well, if it would help. Though it would be nice to have -some- way to kick the CPU in that case. Thoughts?
Quoted 10 lines
Quoted 4 lines
2) Remote wake up done but we haven't yet received the schedule IPI.
3) Non IPI remote wakeup you're referring above, I'm not sure what you mean though.
Well there's two ways of doing remote wakeups, one is doing the wakeup from the waking cpu and sending an IPI over to reschedule iff you need wakeup-preemption, the other is queueing the task remotely and sending an IPI to do the wakeup on the remote cpu.
Quoted 7 lines
The former has the problem, the latter not.
See
ttwu_queue().We could of course mandate that all remote wakeups to special nohz cpus get queued. That would just leave us with RCU and it would simply not send resched IPIs to extended quiescent CPUs anyway, right?
Indeed it never has done this, at least not in the absence of bugs.
Thanx, Paul
Quoted 18 lines
So at that point all return to user
schedule()calls havenr_running> 1 and the tick is running and RCU is not in extended quiescent state. Since either we hadnr_running> 1 and pre and post state are the same, or we hadnr_running== 1 and we just got a fresh wakeup pushing it to 2, the wakeup will have executed on our cpu and have re-started the tick and kicked RCU into active gear again.We cannot hit return to user
schedule()withnr_running== 0, simply because in that case there's no userspace to return to, only the idle thread and that's very much not userspace :-)Hmm ?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
PATCH 12/26 x86: Use the new schedule_user API on userspace preemption
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This way we can exit the RCU extended quiescent state before we schedule a new task from irq/exception exit.
--- arch/x86/kernel/entry_64.S | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/kernel/entry_64.S +4 -4
@@ -565,7 +565,7 @@ sysret_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi jmp sysret_check
@@ -678,7 +678,7 @@ int_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF
@@ -974,7 +974,7 @@ retint_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi GET_THREAD_INFO(%rcx) DISABLE_INTERRUPTS(CLBR_NONE)
@@ -1449,7 +1449,7 @@ paranoid_userspace: paranoid_schedule: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_ANY) - call schedule + call schedule_user DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF jmp paranoid_userspace
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 12/26] x86: Use the new schedule_user API on userspace preemption
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
This way we can exit the RCU extended quiescent state before we schedule a new task from irq/exception exit.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 46 lines
arch/x86/kernel/entry_64.S | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)arch/x86/kernel/entry_64.S +4
-4@@ -565,7 +565,7 @@ sysret_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi jmp sysret_check@@ -678,7 +678,7 @@ int_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi DISABLE_INTERRUPTS(CLBR_NONE) TRACE_IRQS_OFF@@ -974,7 +974,7 @@ retint_careful: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_NONE) pushq_cfi %rdi - call schedule + call schedule_user popq_cfi %rdi GET_THREAD_INFO(%rcx) DISABLE_INTERRUPTS(CLBR_NONE)@@ -1449,7 +1449,7 @@ paranoid_userspace: paranoid_schedule: TRACE_IRQS_ON ENABLE_INTERRUPTS(CLBR_ANY) - call schedule + call schedule_user DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF jmp paranoid_userspaceSignature · 1 line
1.7.8
PATCH 13/26 x86: Exit RCU extended QS on notify resume
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
do_notify_resume() may be called on irq or exception exit. But at that time the exception has already called rcu_user_enter() and the irq has already called rcu_irq_exit().
Since it can use RCU read side critical section, we must call rcu_user_exit() before doing anything there. Then we must call back rcu_user_enter() after this function because we know we are going to userspace from there.
This complete support for userspace RCU extended quiescent state in x86-64.
--- arch/x86/Kconfig | 1 + arch/x86/kernel/signal.c | 4 ++++ 2 files changed, 5 insertions(+), 0 deletions(-)
arch/x86/Kconfig +1 -0
@@ -97,6 +97,7 @@ config X86 select KTIME_SCALAR if X86_32 select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER + select HAVE_RCU_USER_QS if X86_64 config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS || UPROBES)
arch/x86/kernel/signal.c +4 -0
@@ -779,6 +779,8 @@ static void do_signal(struct pt_regs *regs)void do_notify_resume (struct pt_regs *regs,void *unused,__u32 thread_info_flags) { +rcu_user_exit (); +#ifdef CONFIG_X86_MCE /* notify userspace of pending MCEs */ if (thread_info_flags & _TIF_MCE_NOTIFY)
@@ -804,6 +806,8 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32 thread_info_flags)#ifdef CONFIG_X86_32 clear_thread_flag (TIF_IRET );#endif /* CONFIG_X86_32 */ + +rcu_user_enter (); }void signal_fault (struct pt_regs *regs,void __user *frame,char *where)
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 13/26] x86: Exit RCU extended QS on notify resume
Quoted 26 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
do_notify_resume()may be called on irq or exception exit. But at that time the exception has already calledrcu_user_enter()and the irq has already calledrcu_irq_exit().Since it can use RCU read side critical section, we must call
rcu_user_exit()before doing anything there. Then we must call backrcu_user_enter()after this function because we know we are going to userspace from there.This complete support for userspace RCU extended quiescent state in x86-64.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 41 lines
arch/x86/Kconfig | 1 + arch/x86/kernel/signal.c | 4 ++++ 2 files changed, 5 insertions(+), 0 deletions(-)arch/x86/Kconfig +1
-0@@ -97,6 +97,7 @@ config X86 select KTIME_SCALAR if X86_32 select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER + select HAVE_RCU_USER_QS if X86_64 config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS || UPROBES)arch/x86/kernel/signal.c +4
-0@@ -779,6 +779,8 @@ static void do_signal(struct pt_regs *regs)void do_notify_resume (struct pt_regs *regs,void *unused,__u32 thread_info_flags) { +rcu_user_exit (); +#ifdef CONFIG_X86_MCE /* notify userspace of pending MCEs */ if (thread_info_flags & _TIF_MCE_NOTIFY)@@ -804,6 +806,8 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32 thread_info_flags)#ifdef CONFIG_X86_32 clear_thread_flag (TIF_IRET );#endif /* CONFIG_X86_32 */ + +rcu_user_enter (); }void signal_fault (struct pt_regs *regs,void __user *frame,char *where)Signature · 1 line
1.7.8
PATCH 14/26 rcu: Userspace RCU extended QS selftest
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Provide a config option that enables the userspace RCU extended quiescent state on every CPUs by default.
This is for testing purpose.
--- init/Kconfig | 8 ++++++++ kernel/rcutree.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)
init/Kconfig +8 -0
@@ -451,6 +451,14 @@ config RCU_USER_QS excluded from the global RCU state machine and thus doesn't to keep the timer tick on for RCU. +config RCU_USER_QS_FORCE + bool "Force userspace extended QS by default" + depends on RCU_USER_QS + help + Set the hooks in user/kernel boundaries by default in order to + test this feature that treats userspace as an extended quiescent + state until we have a real user like a full adaptive nohz option. + config RCU_FANOUT int "Tree-based hierarchical RCU fanout value" range 2 64 if 64BIT
kernel/rcutree.c +1 -1
@@ -210,7 +210,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = { .dynticks_nesting =DYNTICK_TASK_EXIT_IDLE , .dynticks =ATOMIC_INIT (1 ), -#ifdef CONFIG_RCU_USER_QS +#if defined(CONFIG_RCU_USER_QS) && !defined(CONFIG_RCU_USER_QS_FORCE) .ignore_user_qs = true, #endif };
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 14/26] rcu: Userspace RCU extended QS selftest
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
Provide a config option that enables the userspace RCU extended quiescent state on every CPUs by default.
This is for testing purpose.
Signed-off-by Frederic Weisbecker <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Cc H. Peter Anvin <[email protected]>Cc Ingo Molnar <[email protected]>Cc Josh Triplett <[email protected]>
Quoted 8 lines
Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>
Quoted 39 lines
init/Kconfig | 8 ++++++++ kernel/rcutree.c | 2 +- 2 files changed, 9 insertions(+), 1 deletions(-)init/Kconfig +8
-0@@ -451,6 +451,14 @@ config RCU_USER_QS excluded from the global RCU state machine and thus doesn't to keep the timer tick on for RCU. +config RCU_USER_QS_FORCE + bool "Force userspace extended QS by default" + depends on RCU_USER_QS + help + Set the hooks in user/kernel boundaries by default in order to + test this feature that treats userspace as an extended quiescent + state until we have a real user like a full adaptive nohz option. + config RCU_FANOUT int "Tree-based hierarchical RCU fanout value" range 2 64 if 64BITkernel/rcutree.c +1
-1@@ -210,7 +210,7 @@ EXPORT_SYMBOL_GPL(rcu_note_context_switch); DEFINE_PER_CPU(struct rcu_dynticks, rcu_dynticks) = { .dynticks_nesting =DYNTICK_TASK_EXIT_IDLE , .dynticks =ATOMIC_INIT (1 ), -#ifdef CONFIG_RCU_USER_QS +#if defined(CONFIG_RCU_USER_QS) && !defined(CONFIG_RCU_USER_QS_FORCE) .ignore_user_qs = true, #endif };Signature · 1 line
1.7.8
PATCH 15/26 alpha: Fix preemption handling in idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
cpu_idle() is called on the boot CPU by the init code with preemption disabled. But the cpu_idle() function in alpha doesn't handle this when it calls schedule() directly.
Fix it by converting it into schedule_preempt_disabled().
Also disable preemption before calling cpu_idle() from secondary CPU entry code to stay consistent with this state.
--- arch/alpha/kernel/process.c | 3 ++- arch/alpha/kernel/smp.c | 1 + 2 files changed, 3 insertions(+), 1 deletions(-)
arch/alpha/kernel/process.c +2 -1
@@ -56,7 +56,8 @@ cpu_idle(void)while (!need_resched ())cpu_relax (); -schedule (); + +schedule_preempt_disabled (); } }
arch/alpha/kernel/smp.c +1 -0
@@ -166,6 +166,7 @@ smp_callin(void)DBGS (("smp_callin: commencing CPU %d current %p active_mm %p\n" , cpuid, current, current->active_mm)); +preempt_disable ();/* Do nothing. */ cpu_idle (); }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 15/26] alpha: Fix preemption handling in idle loop
Quoted 12 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
cpu_idle()is called on the boot CPU by the init code with preemption disabled. But thecpu_idle()function in alpha doesn't handle this when it callsschedule()directly.Fix it by converting it into
schedule_preempt_disabled().Also disable preemption before calling
cpu_idle()from secondary CPU entry code to stay consistent with this state.
Quoted 7 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Richard Henderson <[email protected]>Cc Ivan Kokshaysky <[email protected]>Cc Matt Turner <[email protected]>Cc alpha <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Michael Cree <[email protected]>
Quoted 33 lines
arch/alpha/kernel/process.c | 3 ++- arch/alpha/kernel/smp.c | 1 + 2 files changed, 3 insertions(+), 1 deletions(-)arch/alpha/kernel/process.c +2
-1@@ -56,7 +56,8 @@ cpu_idle(void)while (!need_resched ())cpu_relax (); -schedule (); + +schedule_preempt_disabled (); } }arch/alpha/kernel/smp.c +1
-0@@ -166,6 +166,7 @@ smp_callin(void)DBGS (("smp_callin: commencing CPU %d current %p active_mm %p\n" , cpuid, current, current->active_mm)); +preempt_disable ();/* Do nothing. */ cpu_idle (); }Signature · 1 line
1.7.8
PATCH 16/26 alpha: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the Alpha's idle loop.
--- arch/alpha/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/alpha/kernel/process.c +3 -0
@@ -28,6 +28,7 @@#include <linux/tty.h> #include <linux/console.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/reg.h> #include <asm/uaccess.h>
@@ -54,9 +55,11 @@ cpu_idle(void)/* FIXME -- EV6 and LCA45 know how to power down the CPU. */ +rcu_idle_enter ();while (!need_resched ())cpu_relax (); +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 16/26] alpha: Add missing RCU idle APIs on idle loop
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the Alpha's idle loop.
Quoted 9 lines
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Richard Henderson <[email protected]>Cc Ivan Kokshaysky <[email protected]>Cc Matt Turner <[email protected]>Cc alpha <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Michael Cree <[email protected]>Cc 3.2.x.. <[email protected]>
Quoted 30 lines
arch/alpha/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/alpha/kernel/process.c +3
-0@@ -28,6 +28,7 @@#include <linux/tty.h> #include <linux/console.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/reg.h> #include <asm/uaccess.h> @@ -54,9 +55,11 @@ cpu_idle(void)/* FIXME -- EV6 and LCA45 know how to power down the CPU. */ +rcu_idle_enter ();while (!need_resched ())cpu_relax (); +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 17/26 cris: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the Cris's idle loop.
--- arch/cris/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/cris/kernel/process.c +3 -0
@@ -25,6 +25,7 @@#include <linux/elfcore.h> #include <linux/mqueue.h> #include <linux/reboot.h> +#include <linux/rcupdate.h> //#define DEBUG
@@ -74,6 +75,7 @@ void cpu_idle (void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void );/*
@@ -86,6 +88,7 @@ void cpu_idle (void) idle = default_idle;idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 17/26] cris: Add missing RCU idle APIs on idle loop
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the Cris's idle loop.
Quoted 7 lines
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Mikael Starvik <[email protected]>Cc Jesper Nilsson <[email protected]>Cc Cris <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 34 lines
arch/cris/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/cris/kernel/process.c +3
-0@@ -25,6 +25,7 @@#include <linux/elfcore.h> #include <linux/mqueue.h> #include <linux/reboot.h> +#include <linux/rcupdate.h> //#define DEBUG @@ -74,6 +75,7 @@ void cpu_idle (void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void );/* @@ -86,6 +88,7 @@ void cpu_idle (void) idle = default_idle;idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 18/26 frv: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the Frv's idle loop.
--- arch/frv/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/frv/kernel/process.c +3 -0
@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/interrupt.h> #include <linux/pagemap.h> +#include <linux/rcupdate.h> #include <asm/asm-offsets.h> #include <asm/uaccess.h>
@@ -69,12 +70,14 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {check_pgt_cache ();if (!frv_dma_inprogress && idle)idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 18/26] frv: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 7 lines
Add this missing pair of calls in the Frv's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc David Howells <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 33 lines
arch/frv/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/frv/kernel/process.c +3
-0@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/interrupt.h> #include <linux/pagemap.h> +#include <linux/rcupdate.h> #include <asm/asm-offsets.h> #include <asm/uaccess.h> @@ -69,12 +70,14 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {check_pgt_cache ();if (!frv_dma_inprogress && idle)idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); }Signature · 1 line
1.7.8
PATCH 19/26 h8300: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the h8300's idle loop.
--- arch/h8300/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/h8300/kernel/process.c +3 -0
@@ -36,6 +36,7 @@#include <linux/reboot.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/traps.h>
@@ -78,8 +79,10 @@ void (*idle)(void) = default_idle;void cpu_idle (void ) {while (1 ) { +rcu_idle_enter ();while (!need_resched ())idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 19/26] h8300: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 7 lines
Add this missing pair of calls in the h8300's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Yoshinori Sato <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 29 lines
arch/h8300/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/h8300/kernel/process.c +3
-0@@ -36,6 +36,7 @@#include <linux/reboot.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/traps.h> @@ -78,8 +79,10 @@ void (*idle)(void) = default_idle;void cpu_idle (void ) {while (1 ) { +rcu_idle_enter ();while (!need_resched ())idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 20/26 m32r: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the m32r's idle loop.
--- arch/m32r/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/m32r/kernel/process.c +3 -0
@@ -26,6 +26,7 @@#include <linux/ptrace.h> #include <linux/unistd.h> #include <linux/hardirq.h> +#include <linux/rcupdate.h> #include <asm/io.h> #include <asm/uaccess.h>
@@ -82,6 +83,7 @@ void cpu_idle (void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void ) = pm_idle;
@@ -90,6 +92,7 @@ void cpu_idle (void)idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 20/26] m32r: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 7 lines
Add this missing pair of calls in the m32r's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Hirokazu Takata <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 34 lines
arch/m32r/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/m32r/kernel/process.c +3
-0@@ -26,6 +26,7 @@#include <linux/ptrace.h> #include <linux/unistd.h> #include <linux/hardirq.h> +#include <linux/rcupdate.h> #include <asm/io.h> #include <asm/uaccess.h> @@ -82,6 +83,7 @@ void cpu_idle (void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void ) = pm_idle;@@ -90,6 +92,7 @@ void cpu_idle (void)idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 21/26 m68k: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the m68k's idle loop.
--- arch/m68k/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/m68k/kernel/process.c +3 -0
@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/init_task.h> #include <linux/mqueue.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/traps.h>
@@ -75,8 +76,10 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
PATCH 21/26 m68k: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the m68k's idle loop.
--- arch/m68k/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/m68k/kernel/process.c +3 -0
@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/init_task.h> #include <linux/mqueue.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/traps.h>
@@ -75,8 +76,10 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 21/26] m68k: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 8 lines
Add this missing pair of calls in the m68k's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc m68k <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 29 lines
arch/m68k/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/m68k/kernel/process.c +3
-0@@ -25,6 +25,7 @@#include <linux/reboot.h> #include <linux/init_task.h> #include <linux/mqueue.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/traps.h> @@ -75,8 +76,10 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 22/26 mn10300: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the mn10300's idle loop.
--- arch/mn10300/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/mn10300/kernel/process.c +3 -0
@@ -25,6 +25,7 @@#include <linux/err.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/pgtable.h> #include <asm/io.h>
@@ -107,6 +108,7 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ for (;;) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void );
@@ -121,6 +123,7 @@ void cpu_idle(void) }idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 22/26] mn10300: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 8 lines
Add this missing pair of calls in the mn10300's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc David Howells <[email protected]>Cc Koichi Yasutake <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 34 lines
arch/mn10300/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/mn10300/kernel/process.c +3
-0@@ -25,6 +25,7 @@#include <linux/err.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/uaccess.h> #include <asm/pgtable.h> #include <asm/io.h> @@ -107,6 +108,7 @@ void cpu_idle(void) {/* endless idle loop with no priority at all */ for (;;) { +rcu_idle_enter ();while (!need_resched ()) {void (*idle)(void );@@ -121,6 +123,7 @@ void cpu_idle(void) }idle (); } +rcu_idle_exit ();schedule_preempt_disabled (); }Signature · 1 line
1.7.8
PATCH 23/26 parisc: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the parisc's idle loop.
--- arch/parisc/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/parisc/kernel/process.c +3 -0
@@ -48,6 +48,7 @@#include <linux/unistd.h> #include <linux/kallsyms.h> #include <linux/uaccess.h> +#include <linux/rcupdate.h> #include <asm/io.h> #include <asm/asm-offsets.h>
@@ -69,8 +70,10 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())barrier (); +rcu_idle_exit ();schedule_preempt_disabled ();check_pgt_cache (); }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 23/26] parisc: Add missing RCU idle APIs on idle loop
Quoted 19 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the parisc's idle loop.
Quoted 7 lines
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc James E.J. Bottomley <[email protected]>Cc Helge Deller <[email protected]>Cc Parisc <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 29 lines
arch/parisc/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/parisc/kernel/process.c +3
-0@@ -48,6 +48,7 @@#include <linux/unistd.h> #include <linux/kallsyms.h> #include <linux/uaccess.h> +#include <linux/rcupdate.h> #include <asm/io.h> #include <asm/asm-offsets.h> @@ -69,8 +70,10 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())barrier (); +rcu_idle_exit ();schedule_preempt_disabled ();check_pgt_cache (); }Signature · 1 line
1.7.8
PATCH 24/26 score: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the scores's idle loop.
--- arch/score/kernel/process.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
arch/score/kernel/process.c +3 -1
@@ -27,6 +27,7 @@#include <linux/reboot.h> #include <linux/elfcore.h> #include <linux/pm.h> +#include <linux/rcupdate.h> void (*pm_power_off)(void );EXPORT_SYMBOL (pm_power_off);
@@ -50,9 +51,10 @@ void __noreturn cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())barrier (); - +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 24/26] score: Add missing RCU idle APIs on idle loop
Quoted 14 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
Quoted 4 lines
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the scores's idle loop.
s/scores's/score/ or s/the scores's/score's/
Quoted 6 lines
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Chen Liqin <[email protected]>Cc Lennox Wu <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
With the fix above,
Quoted 30 lines
arch/score/kernel/process.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)arch/score/kernel/process.c +3
-1@@ -27,6 +27,7 @@#include <linux/reboot.h> #include <linux/elfcore.h> #include <linux/pm.h> +#include <linux/rcupdate.h> void (*pm_power_off)(void );EXPORT_SYMBOL (pm_power_off);@@ -50,9 +51,10 @@ void __noreturn cpu_idle(void) {/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())barrier (); - +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 25/26 xtensa: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays, rcu_idle_enter() and rcu_idle_exit() must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.
This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Add this missing pair of calls in the xtensa's idle loop.
--- arch/xtensa/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/xtensa/kernel/process.c +3 -0
@@ -31,6 +31,7 @@#include <linux/mqueue.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/pgtable.h> #include <asm/uaccess.h>
@@ -110,8 +111,10 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())platform_idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 25/26] xtensa: Add missing RCU idle APIs on idle loop
Quoted 17 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
In the old times, the whole idle task was considered as an RCU quiescent state. But as RCU became more and more successful overtime, some RCU read side critical section have been added even in the code of some architectures idle tasks, for tracing for example.
So nowadays,
rcu_idle_enter()andrcu_idle_exit()must be called by the architecture to tell RCU about the part in the idle loop that doesn't make use of rcu read side critical sections, typically the part that puts the CPU in low power mode.This is necessary for RCU to find the quiescent states in idle in order to complete grace periods.
Quoted 7 lines
Add this missing pair of calls in the xtensa's idle loop.
Reported-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Chris Zankel <[email protected]>Cc 3.2.x.. <[email protected]>Cc Paul E. McKenney <[email protected]>
Quoted 29 lines
arch/xtensa/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/xtensa/kernel/process.c +3
-0@@ -31,6 +31,7 @@#include <linux/mqueue.h> #include <linux/fs.h> #include <linux/slab.h> +#include <linux/rcupdate.h> #include <asm/pgtable.h> #include <asm/uaccess.h> @@ -110,8 +111,10 @@ void cpu_idle(void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();while (!need_resched ())platform_idle (); +rcu_idle_exit ();schedule_preempt_disabled (); } }Signature · 1 line
1.7.8
PATCH 26/26 ia64: Add missing RCU idle APIs on idle loop
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Traditionally, the entire idle task served as an RCU quiescent state. But when RCU read side critical sections started appearing within the idle loop, this traditional strategy became untenable. The fix was to create new RCU APIs named rcu_idle_enter() and rcu_idle_exit(), which must be called by each architecture's idle loop so that RCU can tell when it is safe to ignore a given idle CPU.
Unfortunately, this fix was never applied to ia64, a shortcoming remedied by this commit.
Reported by: Tony Luck <[email protected]>
Tested by: Tony Luck <[email protected]>
--- arch/ia64/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
arch/ia64/kernel/process.c +3 -0
@@ -29,6 +29,7 @@#include <linux/kdebug.h> #include <linux/utsname.h> #include <linux/tracehook.h> +#include <linux/rcupdate.h> #include <asm/cpu.h> #include <asm/delay.h>
@@ -279,6 +280,7 @@ cpu_idle (void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();if (can_do_pal_halt) {current_thread_info ()->status &= ~TS_POLLING ;/*
@@ -309,6 +311,7 @@ cpu_idle (void)normal_xtp ();#endif } +rcu_idle_exit ();schedule_preempt_disabled ();check_pgt_cache ();if (cpu_is_offline (cpu))
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 26/26] ia64: Add missing RCU idle APIs on idle loop
Quoted 9 lines · Paul E. McKenney
From: "Paul E. McKenney" <[email protected]>
Traditionally, the entire idle task served as an RCU quiescent state. But when RCU read side critical sections started appearing within the idle loop, this traditional strategy became untenable. The fix was to create new RCU APIs named
rcu_idle_enter()andrcu_idle_exit(), which must be called by each architecture's idle loop so that RCU can tell when it is safe to ignore a given idle CPU.
Quoted 7 lines
Unfortunately, this fix was never applied to ia64, a shortcoming remedied by this commit.
Reported by: Tony Luck <[email protected]>
Signed-off-by Paul E. McKenney <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Tested by: Tony Luck <[email protected]>
Quoted 34 lines
arch/ia64/kernel/process.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)arch/ia64/kernel/process.c +3
-0@@ -29,6 +29,7 @@#include <linux/kdebug.h> #include <linux/utsname.h> #include <linux/tracehook.h> +#include <linux/rcupdate.h> #include <asm/cpu.h> #include <asm/delay.h> @@ -279,6 +280,7 @@ cpu_idle (void)/* endless idle loop with no priority at all */ while (1 ) { +rcu_idle_enter ();if (can_do_pal_halt) {current_thread_info ()->status &= ~TS_POLLING ;/* @@ -309,6 +311,7 @@ cpu_idle (void)normal_xtp ();#endif } +rcu_idle_exit ();schedule_preempt_disabled ();check_pgt_cache ();if (cpu_is_offline (cpu))Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 01/26] rcu: New rcu_user_enter() and rcu_user_exit() APIs
In reply to Paul E. McKenneyQuoted 26 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
RCU currently insists that only idle tasks can enter RCU idle mode, which prohibits an adaptive tickless kernel (AKA nohz cpusets), which in turn would mean that usermode execution would always take scheduling-clock interrupts, even when there is only one task runnable on the CPU in question.
This commit therefore adds
rcu_user_enter()andrcu_user_exit(), which allow non-idle tasks to enter RCU idle mode. These are quite similar torcu_idle_enter()andrcu_idle_exit(), respectively, except that they omit the idle-task checks.[ Updated to use "user" flag rather than separate check functions. ]
Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>
Quoted 8 lines
Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>
A few suggestions below: an optional microoptimization and some bugfixes. With the bugfixes, and with or without the microoptimization:
Quoted 2 lines
kernel/rcutree.c +0
-0
[...]
kernel/rcutree.c static int rcu_implicit_offline_qs(struct rcu_data *rdp) · line 359
Quoted 7 lines
kernel/rcutree.c +3
-2-static void rcu_idle_enter_common (struct rcu_dynticks *rdtp,long long oldval) +static void rcu_eqs_enter_common (struct rcu_dynticks *rdtp,long long oldval, +bool user) {trace_rcu_dyntick ("Start" , oldval,0 ); -if (!is_idle_task (current)) { +if (!is_idle_task (current) && !user) {
Microoptimization: putting the !user check first (here and in the exit function) would allow the compiler to partially inline rcu_eqs_*_common into the two trivial wrappers and constant-fold away the test for !user.
kernel/rcutree.c void rcu_idle_enter(void) · line 424
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_idle_enter (void ) +{ +rcu_eqs_enter (0 ); +}
s/0/false/
kernel/rcutree.c void rcu_idle_enter(void) · line 438
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_user_enter (void ) +{ +rcu_eqs_enter (1 ); +}
s/1/true/
kernel/rcutree.c static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval) · line 493
Quoted 12 lines
kernel/rcutree.c +3
-2-static void rcu_idle_exit_common (struct rcu_dynticks *rdtp,long long oldval) +static void rcu_eqs_exit_common (struct rcu_dynticks *rdtp,long long oldval, +int user) {smp_mb__before_atomic_inc ();/* Force ordering w/previous sojourn. */ atomic_inc (&rdtp->dynticks);@@ -464,7 +490,7 @@ static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval)WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 ));rcu_cleanup_after_idle (smp_processor_id ());trace_rcu_dyntick ("End" , oldval, rdtp->dynticks_nesting); -if (!is_idle_task (current)) { +if (!is_idle_task (current) && !user) {
Same micro-optimization as the enter function.
kernel/rcutree.c void rcu_idle_exit(void) · line 541
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_idle_exit (void ) +{ +rcu_eqs_exit (0 ); +}
s/0/false/
kernel/rcutree.c void rcu_idle_exit(void) · line 553
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_user_exit (void ) +{ +rcu_eqs_exit (1 ); +}
s/1/true/
kernel/rcutree.c void rcu_irq_enter(void) · line 589
Quoted 6 lines
kernel/rcutree.c +1
-1@@ -539,7 +586,7 @@ void rcu_irq_enter(void)if (oldval)trace_rcu_dyntick ("++=" , oldval, rdtp->dynticks_nesting);else -rcu_idle_exit_common (rdtp, oldval); +rcu_eqs_exit_common (rdtp, oldval,1 );
s/1/true/, and likewise in rcu_irq_exit.
- Josh Triplett
Re: [PATCH tip/core/rcu 01/26] rcu: New rcu_user_enter() and rcu_user_exit() APIs
Quoted 36 lines · Josh Triplett
Quoted 26 lines · Paul E. McKenney
From: Frederic Weisbecker <[email protected]>
RCU currently insists that only idle tasks can enter RCU idle mode, which prohibits an adaptive tickless kernel (AKA nohz cpusets), which in turn would mean that usermode execution would always take scheduling-clock interrupts, even when there is only one task runnable on the CPU in question.
This commit therefore adds
rcu_user_enter()andrcu_user_exit(), which allow non-idle tasks to enter RCU idle mode. These are quite similar torcu_idle_enter()andrcu_idle_exit(), respectively, except that they omit the idle-task checks.[ Updated to use "user" flag rather than separate check functions. ]
Signed-off-by Frederic Weisbecker <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Cc Alessio Igor Bogani <[email protected]>Cc Andrew Morton <[email protected]>Cc Avi Kivity <[email protected]>Cc Chris Metcalf <[email protected]>Cc Christoph Lameter <[email protected]>Cc Daniel Lezcano <[email protected]>Cc Geoff Levand <[email protected]>Cc Gilad Ben Yossef <[email protected]>Cc Hakan Akkan <[email protected]>Quoted 8 lines
Cc Ingo Molnar <[email protected]>Cc Kevin Hilman <[email protected]>Cc Max Krasnyansky <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Steven Rostedt <[email protected]>Cc Sven-Thorsten Dietrich <[email protected]>Cc Thomas Gleixner <[email protected]>
Quoted 2 lines
A few suggestions below: an optional microoptimization and some bugfixes. With the bugfixes, and with or without the microoptimization:
Good catches! Due to conflicts with later commits, I added these as a separate commit.
Thanx, Paul
Quoted 71 lines
Reviewed-by Josh Triplett <[email protected]>Quoted 2 lines
kernel/rcutree.c +0
-0[...]
kernel/rcutree.c static int rcu_implicit_offline_qs(struct rcu_data *rdp) · line 359
Quoted 7 lines
kernel/rcutree.c +3
-2-static void rcu_idle_enter_common (struct rcu_dynticks *rdtp,long long oldval) +static void rcu_eqs_enter_common (struct rcu_dynticks *rdtp,long long oldval, +bool user) {trace_rcu_dyntick ("Start" , oldval,0 ); -if (!is_idle_task (current)) { +if (!is_idle_task (current) && !user) {Microoptimization: putting the !user check first (here and in the exit function) would allow the compiler to partially inline rcu_eqs_*_common into the two trivial wrappers and constant-fold away the test for !user.
kernel/rcutree.c void rcu_idle_enter(void) · line 424
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_idle_enter (void ) +{ +rcu_eqs_enter (0 ); +}s/0/false/
kernel/rcutree.c void rcu_idle_enter(void) · line 438
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_user_enter (void ) +{ +rcu_eqs_enter (1 ); +}s/1/true/
kernel/rcutree.c static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval) · line 493
Quoted 12 lines
kernel/rcutree.c +3
-2-static void rcu_idle_exit_common (struct rcu_dynticks *rdtp,long long oldval) +static void rcu_eqs_exit_common (struct rcu_dynticks *rdtp,long long oldval, +int user) {smp_mb__before_atomic_inc ();/* Force ordering w/previous sojourn. */ atomic_inc (&rdtp->dynticks);@@ -464,7 +490,7 @@ static void rcu_idle_exit_common(struct rcu_dynticks *rdtp, long long oldval)WARN_ON_ONCE (!(atomic_read (&rdtp->dynticks) &0x1 ));rcu_cleanup_after_idle (smp_processor_id ());trace_rcu_dyntick ("End" , oldval, rdtp->dynticks_nesting); -if (!is_idle_task (current)) { +if (!is_idle_task (current) && !user) {Same micro-optimization as the enter function.
kernel/rcutree.c void rcu_idle_exit(void) · line 541
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_idle_exit (void ) +{ +rcu_eqs_exit (0 ); +}s/0/false/
kernel/rcutree.c void rcu_idle_exit(void) · line 553
Quoted 4 lines
kernel/rcutree.c +4
-0+void rcu_user_exit (void ) +{ +rcu_eqs_exit (1 ); +}s/1/true/
kernel/rcutree.c void rcu_irq_enter(void) · line 589
Quoted 6 lines
kernel/rcutree.c +1
-1@@ -539,7 +586,7 @@ void rcu_irq_enter(void)if (oldval)trace_rcu_dyntick ("++=" , oldval, rdtp->dynticks_nesting);else -rcu_idle_exit_common (rdtp, oldval); +rcu_eqs_exit_common (rdtp, oldval,1 );s/1/true/, and likewise in
rcu_irq_exit.- Josh Triplett