RCU-tasks implementation
Subsystems (9)
v4 cover RCU-tasks implementation
Hello!
This series provides v4 of a prototype of an RCU-tasks implementation, which has been requested to assist with tramopoline removal. This flavor of RCU is task-based rather than CPU-based, and has voluntary context switch, usermode execution, and the idle loops as its only quiescent states. This selection of quiescent states ensures that at the end of a grace period, there will no longer be any tasks depending on a trampoline that was removed before the beginning of that grace period. This works because such trampolines do not contain function calls, do not contain voluntary context switches, do not switch to usermode, and do not switch to idle.
The patches in this series are as follows:
1. Adds the basic call_rcu_tasks() functionality.
2. Provides cond_resched_rcu_qs() to force quiescent states, including
RCU-tasks quiescent states, in long loops.
3. Adds synchronous APIs: synchronize_rcu_tasks() and
rcu_barrier_tasks().
4. Handle the possibility of tasks being preempted for extended
periods of time after being removed from the task list.
5. Adds GPL exports for the above APIs, courtesy of Steven Rostedt.
6. Adds rcutorture tests for RCU-tasks.
7. Adds RCU-tasks test cases to rcutorture scripting.
8. Adds stall-warning checks for RCU-tasks.
9. Improves RCU-tasks energy efficiency by replacing polling with
wait/wakeup.
10. Document RCU-tasks stall-warning messages.
11. Defer rcu_tasks_kthread() creation until first call_rcu_tasks()
to avoid populating systems with unneeded kthreads.
Changes from v3:
o Add do_exit() SRCU hooks to handle tasks being preempted after
having removed themselves from the task list. The need for this was pointed out by Oleg Nesterov, and the implmentation suggested by Lai Jiangshan.
o Create rcu_tasks_kthread only if call_rcu_tasks() is invoked.
Changes from v2:
o Use get_task_struct() instead of do_exit() hooks to synchronize
with exiting tasks, as suggested by Lai Jiangshan.
o Add checks of ->on_rq to the grace-period-wait polling, again
as suggested by Lai Jiangshan.
o Repositioned synchronize_sched() calls and improved their
comments.
Changes from v1:
o The lockdep issue with list locking was finessed by ditching
list locking in favor of having the list manipulated by a single kthread. This change trimmed about 150 highly concurrent lines from the implementation.
o Get rid of the scheduler hooks in favor of polling the
per-task count of voluntary context switches, in response to Peter Zijlstra's concerns about scheduler overhead.
o Passes more aggressive rcutorture runs, which indicates that
an increase in rcutorture's aggression is called for.
o Handled review comments from Peter Zijlstra, Lai Jiangshan,
Frederic Weisbecker, and Oleg Nesterov.
o Added RCU-tasks stall-warning documentation.
Remaining issues include:
o It is not clear that trampolines in functions called from the
idle loop are correctly handled. Or if anyone cares about trampolines in functions called from the idle loop.
o The current implementation does not yet recognize tasks that start
out executing is usermode. Instead, it waits for the next scheduling-clock tick to note them.
o As a result, the current implementation does not handle nohz_full=
CPUs executing tasks running in usermode. There are a couple of possible fixes under consideration.
o If a task is preempted while executing in usermode, the RCU-tasks
grace period will not end until that task resumes. (Is there some reasonable way to determine that a given preempted task was preempted from usermode execution?)
o More about RCU-tasks needs to be added to Documentation/RCU.
o There are probably still bugs.
Thanx, Paul
------------------------------------------------------------------------
b/Documentation/RCU/stallwarn.txt | 33 - b/Documentation/kernel-parameters.txt | 5 b/fs/file.c | 2 b/include/linux/init_task.h | 9 b/include/linux/rcupdate.h | 55 + b/include/linux/sched.h | 23 b/init/Kconfig | 10 b/kernel/exit.c | 3 b/kernel/rcu/rcutorture.c | 54 + b/kernel/rcu/tiny.c | 2 b/kernel/rcu/tree.c | 14 b/kernel/rcu/tree_plugin.h | 2 b/kernel/rcu/update.c | 322 +++++++++- b/mm/mlock.c | 2 b/tools/testing/selftests/rcutorture/configs/rcu/TASKS01 | 7 b/tools/testing/selftests/rcutorture/configs/rcu/TASKS01.boot | 1 b/tools/testing/selftests/rcutorture/configs/rcu/TASKS02 | 6 b/tools/testing/selftests/rcutorture/configs/rcu/TASKS02.boot | 1 18 files changed, 508 insertions(+), 43 deletions(-)
PATCH v4 1/11 rcu: Add call_rcu_tasks()
From: "Paul E. McKenney" <[email protected]>
This commit adds a new RCU-tasks flavor of RCU, which provides call_rcu_tasks(). This RCU flavor's quiescent states are voluntary context switch (not preemption!), userspace execution, and the idle loop. Note that unlike other RCU flavors, these quiescent states occur in tasks, not necessarily CPUs. Includes fixes from Steven Rostedt.
This RCU flavor is assumed to have very infrequent latency-tolerate updaters. This assumption permits significant simplifications, including a single global callback list protected by a single global lock, along with a single linked list containing all tasks that have not yet passed through a quiescent state. If experience shows this assumption to be incorrect, the required additional complexity will be added.
--- include/linux/init_task.h | 9 +++ include/linux/rcupdate.h | 36 ++++++++++ include/linux/sched.h | 23 ++++--- init/Kconfig | 10 +++ kernel/rcu/tiny.c | 2 + kernel/rcu/tree.c | 2 + kernel/rcu/update.c | 172 ++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 243 insertions(+), 11 deletions(-)
include/linux/init_task.h +9 -0
@@ -124,6 +124,14 @@ extern struct group_info init_groups;#else #define INIT_TASK_RCU_PREEMPT (tsk)#endif +#ifdef CONFIG_TASKS_RCU +#define INIT_TASK_RCU_TASKS (tsk) \ + .rcu_tasks_holdout = false, \ + .rcu_tasks_holdout_list = \ + LIST_HEAD_INIT(tsk.rcu_tasks_holdout_list), +#else +#define INIT_TASK_RCU_TASKS (tsk) +#endif extern struct cred init_cred;
@@ -231,6 +239,7 @@ extern struct task_group root_task_group;INIT_FTRACE_GRAPH \ INIT_TRACE_RECURSION \ INIT_TASK_RCU_PREEMPT(tsk) \ + INIT_TASK_RCU_TASKS(tsk) \ INIT_CPUSET_SEQ(tsk) \ INIT_RT_MUTEXES(tsk) \ INIT_VTIME(tsk) \
include/linux/rcupdate.h +36 -0
@@ -197,6 +197,26 @@ void call_rcu_sched(struct rcu_head *head,void synchronize_sched (void ); +/** +* call_rcu_tasks() - Queue an RCU for invocation task-based grace period +* @head: structure to be used for queueing the RCU updates. +* @func: actual callback function to be invoked after the grace period +* +* The callback function will be invoked some time after a full grace +* period elapses, in other words after all currently executing RCU +* read-side critical sections have completed. call_rcu_tasks() assumes +* that the read-side critical sections end at a voluntary context +* switch (not a preemption!), entry into idle, or transition to usermode +* execution. As such, there are no read-side primitives analogous to +* rcu_read_lock() and rcu_read_unlock() because this primitive is intended +* to determine that all tasks have passed through a safe state, not so +* much for data-strcuture synchronization. +* +* See the description of call_rcu() for more detailed information on +* memory ordering guarantees. +*/ +void call_rcu_tasks (struct rcu_head *head,void (*func)(struct rcu_head *head)); +#ifdef CONFIG_PREEMPT_RCU void __rcu_read_lock (void );
@@ -294,6 +314,22 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev,rcu_irq_exit (); \ }while (0 ) +/* +* Note a voluntary context switch for RCU-tasks benefit. This is a +* macro rather than an inline function to avoid #include hell. + */ +#ifdef CONFIG_TASKS_RCU +#define rcu_note_voluntary_context_switch(t) \ + do { \ + preempt_disable(); /* Exclude synchronize_sched(); */ \ + if (ACCESS_ONCE((t)->rcu_tasks_holdout)) \ + ACCESS_ONCE((t)->rcu_tasks_holdout) = 0; \ + preempt_enable(); \ + } while (0) +#else /* #ifdef CONFIG_TASKS_RCU */ +#define rcu_note_voluntary_context_switch (t) do { } while (0) +#endif /* #else #ifdef CONFIG_TASKS_RCU */ +#if defined(CONFIG_DEBUG_LOCK_ALLOC ) || defined(CONFIG_RCU_TRACE ) || defined(CONFIG_SMP )bool __rcu_is_watching (void );#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP) */
include/linux/sched.h +12 -11
@@ -1273,6 +1273,11 @@ struct task_struct {#ifdef CONFIG_RCU_BOOST struct rt_mutex *rcu_boost_mutex;#endif /* #ifdef CONFIG_RCU_BOOST */ +#ifdef CONFIG_TASKS_RCU +unsigned long rcu_tasks_nvcsw; +int rcu_tasks_holdout; +struct list_head rcu_tasks_holdout_list; +#endif /* #ifdef CONFIG_TASKS_RCU */ #if defined(CONFIG_SCHEDSTATS ) || defined(CONFIG_TASK_DELAY_ACCT )struct sched_info sched_info;
@@ -1998,31 +2003,27 @@ extern void task_clear_jobctl_pending(struct task_struct *task, unsigned int mask);#ifdef CONFIG_PREEMPT_RCU -#define RCU_READ_UNLOCK_BLOCKED (1 << 0)/* blocked while in RCU read-side. */ #define RCU_READ_UNLOCK_NEED_QS (1 << 1)/* RCU core needs CPU response. */ +#endif /* #ifdef CONFIG_PREEMPT_RCU */ static inline void rcu_copy_process (struct task_struct *p) { +#ifdef CONFIG_PREEMPT_RCU p->rcu_read_lock_nesting =0 ; p->rcu_read_unlock_special =0 ; -#ifdef CONFIG_TREE_PREEMPT_RCU p->rcu_blocked_node =NULL ; -#endif /* #ifdef CONFIG_TREE_PREEMPT_RCU */ #ifdef CONFIG_RCU_BOOST p->rcu_boost_mutex =NULL ;#endif /* #ifdef CONFIG_RCU_BOOST */ INIT_LIST_HEAD (&p->rcu_node_entry); +#endif /* #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_TASKS_RCU + p->rcu_tasks_holdout = false; +INIT_LIST_HEAD (&p->rcu_tasks_holdout_list); +#endif /* #ifdef CONFIG_TASKS_RCU */ } -#else - -static inline void rcu_copy_process (struct task_struct *p) -{ -} - -#endif -static inline void tsk_restore_flags (struct task_struct *task,unsigned long orig_flags,unsigned long flags) {
init/Kconfig +10 -0
@@ -507,6 +507,16 @@ config PREEMPT_RCU This option enables preemptible-RCU code that is common between the TREE_PREEMPT_RCU and TINY_PREEMPT_RCU implementations. +config TASKS_RCU + bool "Task_based RCU implementation using voluntary context switch" + default n + help + This option enables a task-based RCU implementation that uses + only voluntary context switch (not preemption!), idle, and + user-mode execution as quiescent states. + + If unsure, say N. + config RCU_STALL_COMMON def_bool ( TREE_RCU || TREE_PREEMPT_RCU || RCU_TRACE ) help
kernel/rcu/tiny.c +2 -0
@@ -254,6 +254,8 @@ void rcu_check_callbacks(int cpu, int user)rcu_sched_qs (cpu); else if (!in_softirq()) rcu_bh_qs(cpu); +if (user) +rcu_note_voluntary_context_switch (current); }/*
kernel/rcu/tree.c +2 -0
@@ -2413,6 +2413,8 @@ void rcu_check_callbacks(int cpu, int user)rcu_preempt_check_callbacks (cpu);if (rcu_pending (cpu))invoke_rcu_core (); +if (user) +rcu_note_voluntary_context_switch (current);trace_rcu_utilization (TPS ("End scheduler-tick" )); }
kernel/rcu/update.c +172 -0
181 lines not on this page · raw message
Signature · 1 line
1.8.1.5
PATCH v4 2/11 rcu: Provide cond_resched_rcu_qs() to force quiescent states in long loops
From: "Paul E. McKenney" <[email protected]>
RCU-tasks requires the occasional voluntary context switch from CPU-bound in-kernel tasks. In some cases, this requires instrumenting cond_resched(). However, there is some reluctance to countenance unconditionally instrumenting cond_resched() (see
http://lwn.net/Articles/603252/), so this commit creates a separate cond_resched_rcu_qs() that may be used in place of cond_resched() in locations prone to long-duration in-kernel looping.
This commit currently instruments only RCU-tasks. Future possibilities include also instrumenting RCU, RCU-bh, and RCU-sched in order to reduce IPI usage.
--- fs/file.c | 2 +- include/linux/rcupdate.h | 13 +++++++++++++ kernel/rcu/rcutorture.c | 4 ++-- kernel/rcu/tree.c | 12 ++++++------ kernel/rcu/tree_plugin.h | 2 +- mm/mlock.c | 2 +- 6 files changed, 24 insertions(+), 11 deletions(-)
fs/file.c +1 -1
@@ -367,7 +367,7 @@ static struct fdtable *close_files(struct files_struct * files)struct file * file =xchg (&fdt->fd[i],NULL );if (file) {filp_close (file, files); -cond_resched (); +cond_resched_rcu_qs (); } } i++;
include/linux/rcupdate.h +13 -0
@@ -330,6 +330,19 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev,#define rcu_note_voluntary_context_switch (t) do { } while (0)#endif /* #else #ifdef CONFIG_TASKS_RCU */ +/** +* cond_resched_rcu_qs - Report potential quiescent states to RCU +* +* This macro resembles cond_resched(), except that it is defined to +* report potential quiescent states to RCU-tasks even if the cond_resched() +* machinery were to be shut off, as some advocate for PREEMPT kernels. +*/ +#define cond_resched_rcu_qs () \ +do { \ + rcu_note_voluntary_context_switch(current); \ + cond_resched(); \ +} while (0) +#if defined(CONFIG_DEBUG_LOCK_ALLOC ) || defined(CONFIG_RCU_TRACE ) || defined(CONFIG_SMP )bool __rcu_is_watching (void );#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP) */
kernel/rcu/rcutorture.c +2 -2
@@ -667,7 +667,7 @@ static int rcu_torture_boost(void *arg) } call_rcu_time = jiffies; } -cond_resched (); +cond_resched_rcu_qs ();stutter_wait ("rcu_torture_boost" );if (torture_must_stop ()) goto checkwait;
@@ -1019,7 +1019,7 @@ rcu_torture_reader(void *arg)__this_cpu_inc (rcu_torture_batch[completed]);preempt_enable (); cur_ops->readunlock (idx); -cond_resched (); +cond_resched_rcu_qs ();stutter_wait ("rcu_torture_reader" ); }while (!torture_must_stop ());if (irqreader && cur_ops->irq_capable) {
kernel/rcu/tree.c +6 -6
@@ -1650,7 +1650,7 @@ static int rcu_gp_init(struct rcu_state *rsp) system_state == SYSTEM_RUNNING)udelay (200 );#endif /* #ifdef CONFIG_PROVE_RCU_DELAY */ -cond_resched (); +cond_resched_rcu_qs (); }mutex_unlock (&rsp->onoff_mutex);
@@ -1739,7 +1739,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)/* smp_mb() provided by prior unlock-lock pair. */ nocb +=rcu_future_gp_cleanup (rsp, rnp);raw_spin_unlock_irq (&rnp->lock); -cond_resched (); +cond_resched_rcu_qs (); } rnp =rcu_get_root (rsp);raw_spin_lock_irq (&rnp->lock);
@@ -1788,7 +1788,7 @@ static int __noreturn rcu_gp_kthread(void *arg)/* Locking provides needed memory barrier. */ if (rcu_gp_init (rsp))break ; -cond_resched (); +cond_resched_rcu_qs ();flush_signals (current); trace_rcu_grace_period(rsp->name, ACCESS_ONCE(rsp->gpnum),
@@ -1831,10 +1831,10 @@ static int __noreturn rcu_gp_kthread(void *arg)trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum),TPS ("fqsend" )); -cond_resched (); +cond_resched_rcu_qs (); }else {/* Deal with stray signal. */ -cond_resched (); +cond_resched_rcu_qs ();flush_signals (current); trace_rcu_grace_period(rsp->name, ACCESS_ONCE(rsp->gpnum),
@@ -2437,7 +2437,7 @@ static void force_qs_rnp(struct rcu_state *rsp,struct rcu_node *rnp;rcu_for_each_leaf_node (rsp, rnp) { -cond_resched (); +cond_resched_rcu_qs (); mask =0 ;raw_spin_lock_irqsave (&rnp->lock, flags);smp_mb__after_unlock_lock ();
kernel/rcu/tree_plugin.h +1 -1
@@ -1842,7 +1842,7 @@ static int rcu_oom_notify(struct notifier_block *self,get_online_cpus ();for_each_online_cpu (cpu) {smp_call_function_single (cpu, rcu_oom_notify_cpu,NULL ,1 ); -cond_resched (); +cond_resched_rcu_qs (); }put_online_cpus ();
mm/mlock.c +1 -1
@@ -782,7 +782,7 @@ static int do_mlockall(int flags)/* Ignore errors */ mlock_fixup (vma, &prev, vma->vm_start, vma->vm_end, newflags); -cond_resched (); +cond_resched_rcu_qs (); } out:return 0 ;
Signature · 1 line
1.8.1.5
PATCH v4 3/11 rcu: Add synchronous grace-period waiting for RCU-tasks
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
It turns out to be easier to add the synchronous grace-period waiting functions to RCU-tasks than to work around their absense in rcutorture, so this commit adds them. The key point is that the existence of call_rcu_tasks() means that rcutorture needs an rcu_barrier_tasks().
--- include/linux/rcupdate.h | 2 ++ kernel/rcu/update.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)
include/linux/rcupdate.h +2 -0
@@ -216,6 +216,8 @@ void synchronize_sched(void);* memory ordering guarantees. */ void call_rcu_tasks (struct rcu_head *head,void (*func)(struct rcu_head *head)); +void synchronize_rcu_tasks (void ); +void rcu_barrier_tasks (void );#ifdef CONFIG_PREEMPT_RCU
kernel/rcu/update.c +55 -0
61 lines not on this page · raw message
Signature · 1 line
1.8.1.5
PATCH v4 4/11 rcu: Make TASKS_RCU handle tasks that are almost done exiting
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Once a task has passed exit_notify() in the do_exit() code path, it is no longer on the task lists, and is therefore no longer visible to rcu_tasks_kthread(). This means that an almost-exited task might be preempted while within a trampoline, and this task won't be waited on by rcu_tasks_kthread(). This commit fixes this bug by adding an srcu_struct. An exiting task does srcu_read_lock() just before calling exit_notify(), and does the corresponding srcu_read_unlock() after doing the final preempt_disable(). This means that rcu_tasks_kthread() can do synchronize_srcu() to wait for all mostly-exited tasks to reach their final preempt_disable() region, and then use synchronize_sched() to wait for those tasks to finish exiting.
--- include/linux/rcupdate.h | 3 +++ kernel/exit.c | 3 +++ kernel/rcu/update.c | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+)
include/linux/rcupdate.h +3 -0
@@ -321,6 +321,8 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev,* macro rather than an inline function to avoid #include hell. */ #ifdef CONFIG_TASKS_RCU +#define TASKS_RCU (x) x +extern struct srcu_struct tasks_rcu_exit_srcu;#define rcu_note_voluntary_context_switch (t) \ do { \ preempt_disable();/* Exclude synchronize_sched(); */ \
@@ -329,6 +331,7 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev,preempt_enable (); \ } while (0)#else /* #ifdef CONFIG_TASKS_RCU */ +#define TASKS_RCU (x) do { } while (0)#define rcu_note_voluntary_context_switch (t) do { } while (0)#endif /* #else #ifdef CONFIG_TASKS_RCU */
kernel/exit.c +3 -0
@@ -663,6 +663,7 @@ void do_exit(long code) {struct task_struct *tsk = current;int group_dead; + TASKS_RCU(int tasks_rcu_i);profile_task_exit (tsk);
@@ -772,6 +773,7 @@ void do_exit(long code)*/ flush_ptrace_hw_breakpoint (tsk); +TASKS_RCU (tasks_rcu_i =srcu_read_lock (&tasks_rcu_exit_srcu));exit_notify (tsk, group_dead);proc_exit_connector (tsk);#ifdef CONFIG_NUMA
@@ -811,6 +813,7 @@ void do_exit(long code)if (tsk->nr_dirtied)__this_cpu_add (dirty_throttle_leaks, tsk->nr_dirtied);exit_rcu (); +TASKS_RCU (srcu_read_unlock (&tasks_rcu_exit_srcu, tasks_rcu_i));/* * The setting of TASK_RUNNING by try_to_wake_up() may be delayed
kernel/rcu/update.c +21 -0
@@ -370,6 +370,13 @@ static struct rcu_head *rcu_tasks_cbs_head;static struct rcu_head **rcu_tasks_cbs_tail = &rcu_tasks_cbs_head;static DEFINE_RAW_SPINLOCK (rcu_tasks_cbs_lock); +/* Track exiting tasks in order to allow them to be waited for. */ +DEFINE_SRCU (tasks_rcu_exit_srcu); + +/* Control stall timeouts. Disable with <= 0, otherwise jiffies till stall. */ +static int rcu_task_stall_timeout__read_mostly =HZ *60 *3 ; +module_param (rcu_task_stall_timeout, int,0644 ); +/* Post an RCU-tasks callback. */ void call_rcu_tasks (struct rcu_head *rhp,void (*func)(struct rcu_head *rhp)) {
@@ -522,6 +529,15 @@ static int __noreturn rcu_tasks_kthread(void *arg)rcu_read_unlock ();/* +* Wait for tasks that are in the process of exiting. +* This does only part of the job, ensuring that all +* tasks that were previously exiting reach the point +* where they have disabled preemption, allowing the +* later synchronize_sched() to finish the job. +*/ +synchronize_srcu (&tasks_rcu_exit_srcu); + +/* * Each pass through the following loop scans the list * of holdout tasks, removing any that are no longer * holdouts. When the list is empty, we are done.
@@ -550,6 +566,11 @@ static int __noreturn rcu_tasks_kthread(void *arg)* ->rcu_tasks_holdout accesses to be within the grace * period, avoiding the need for memory barriers for * ->rcu_tasks_holdout accesses. +* +* In addition, this synchronize_sched() waits for exiting +* tasks to complete their final preempt_disable() region +* of execution, cleaning up after the synchronize_srcu() +* above. */ synchronize_sched ();
Signature · 1 line
1.8.1.5
PATCH v4 5/11 rcu: Export RCU-tasks APIs to GPL modules
In reply to Paul E. McKenneyFrom: Steven Rostedt <[email protected]>
This commit exports the RCU-tasks APIs, call_rcu_tasks(), synchronize_rcu_tasks(), and rcu_barrier_tasks(), to GPL-licensed kernel modules.
--- kernel/rcu/update.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/rcu/update.c +2 -0
@@ -433,6 +433,7 @@ void synchronize_rcu_tasks(void)/* Wait for the grace period. */ wait_rcu_gp (call_rcu_tasks); } +EXPORT_SYMBOL_GPL (synchronize_rcu_tasks);/** * rcu_barrier_tasks - Wait for in-flight call_rcu_tasks() callbacks.
@@ -445,6 +446,7 @@ void rcu_barrier_tasks(void)/* There is only one callback queue, so this is easy. ;-) */ synchronize_rcu_tasks (); } +EXPORT_SYMBOL_GPL (rcu_barrier_tasks);/* See if tasks are still holding out, complain if so. */ static void check_holdout_task (struct task_struct *t)
Signature · 1 line
1.8.1.5
PATCH v4 6/11 rcutorture: Add torture tests for RCU-tasks
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds torture tests for RCU-tasks. It also fixes a bug that would segfault for an RCU flavor lacking a callback-barrier function.
--- include/linux/rcupdate.h | 1 + kernel/rcu/rcutorture.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 1 deletion(-)
include/linux/rcupdate.h +1 -0
@@ -55,6 +55,7 @@ enum rcutorture_type {RCU_FLAVOR ,RCU_BH_FLAVOR ,RCU_SCHED_FLAVOR , +RCU_TASKS_FLAVOR ,SRCU_FLAVOR , INVALID_RCU_FLAVOR };
kernel/rcu/rcutorture.c +49 -1
68 lines not on this page · raw message
Signature · 1 line
1.8.1.5
PATCH v4 7/11 rcutorture: Add RCU-tasks test cases
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds the TASKS01 and TASKS02 Kconfig fragments, along with the corresponding TASKS01.boot and TASKS02.boot boot-parameter files specifying that rcutorture test RCU-tasks instead of the default flavor.
--- tools/testing/selftests/rcutorture/configs/rcu/TASKS01 | 7 +++++++ tools/testing/selftests/rcutorture/configs/rcu/TASKS01.boot | 1 + tools/testing/selftests/rcutorture/configs/rcu/TASKS02 | 6 ++++++ tools/testing/selftests/rcutorture/configs/rcu/TASKS02.boot | 1 + 4 files changed, 15 insertions(+) create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TASKS01 create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TASKS01.boot create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TASKS02 create mode 100644 tools/testing/selftests/rcutorture/configs/rcu/TASKS02.boot
tools/testing/selftests/rcutorture/configs/rcu/TASKS01 new file +7 -0
@@ -0,0 +1,7 @@ +CONFIG_SMP=y +CONFIG_NR_CPUS=2 +CONFIG_HOTPLUG_CPU=y +CONFIG_PREEMPT_NONE=n +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=y +CONFIG_TASKS_RCU=y
tools/testing/selftests/rcutorture/configs/rcu/TASKS01.boot new file +1 -0
@@ -0,0 +1 @@ +rcutorture.torture_type=tasks
tools/testing/selftests/rcutorture/configs/rcu/TASKS02 new file +6 -0
@@ -0,0 +1,6 @@ +CONFIG_SMP=n +CONFIG_HOTPLUG_CPU=y +CONFIG_PREEMPT_NONE=y +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=n +CONFIG_TASKS_RCU=y
tools/testing/selftests/rcutorture/configs/rcu/TASKS02.boot new file +1 -0
@@ -0,0 +1 @@ +rcutorture.torture_type=tasks
Signature · 1 line
1.8.1.5
PATCH v4 8/11 rcu: Add stall-warning checks for RCU-tasks
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds a three-minute RCU-tasks stall warning. The actual time is controlled by the boot/sysfs parameter rcu_task_stall_timeout, with values less than or equal to zero disabling the stall warnings. The default value is three minutes, which means that the tasks that have not yet responded will get their stacks dumped every three minutes, until they pass through a voluntary context switch.
--- Documentation/kernel-parameters.txt | 5 +++++ kernel/rcu/update.c | 25 +++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 2 deletions(-)
Documentation/kernel-parameters.txt +5 -0
@@ -2921,6 +2921,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted. rcupdate.rcu_cpu_stall_timeout= [KNL] Set timeout for RCU CPU stall warning messages. + rcupdate.rcu_task_stall_timeout= [KNL] + Set timeout in jiffies for RCU task stall warning + messages. Disable with a value less than or equal + to zero. + rdinit= [KNL] Format: <full_path> Run specified binary instead of /init from the ramdisk,
kernel/rcu/update.c +23 -2
@@ -449,7 +449,8 @@ void rcu_barrier_tasks(void)EXPORT_SYMBOL_GPL (rcu_barrier_tasks);/* See if tasks are still holding out, complain if so. */ -static void check_holdout_task (struct task_struct *t) +static void check_holdout_task (struct task_struct *t, +bool needreport,bool *firstreport) { if (!ACCESS_ONCE(t->rcu_tasks_holdout) || t->rcu_tasks_nvcsw != ACCESS_ONCE(t->nvcsw) ||
@@ -457,7 +458,15 @@ static void check_holdout_task(struct task_struct *t)ACCESS_ONCE (t->rcu_tasks_holdout) =0 ;list_del_rcu (&t->rcu_tasks_holdout_list);put_task_struct (t); +return ; } +if (!needreport) +return ; +if (*firstreport) { +pr_err ("INFO: rcu_tasks detected stalls on tasks:\n" ); + *firstreport = false; + } +sched_show_task (current); }/* RCU-tasks kthread that detects grace periods and invokes callbacks. */
@@ -465,6 +474,7 @@ static int __noreturn rcu_tasks_kthread(void *arg) {unsigned long flags;struct task_struct *g, *t; +unsigned long lastreport;struct rcu_head *list;struct rcu_head *next;LIST_HEAD (rcu_tasks_holdouts);
@@ -544,13 +554,24 @@ static int __noreturn rcu_tasks_kthread(void *arg)* of holdout tasks, removing any that are no longer * holdouts. When the list is empty, we are done. */ + lastreport = jiffies;while (!list_empty (&rcu_tasks_holdouts)) { +bool firstreport; +bool needreport; +int rtst; +schedule_timeout_interruptible (HZ /10 ); + rtst =ACCESS_ONCE (rcu_task_stall_timeout); + needreport = rtst >0 && +time_after (jiffies, lastreport + rtst); +if (needreport) + lastreport = jiffies; + firstreport = true;WARN_ON (signal_pending (current));rcu_read_lock ();list_for_each_entry_rcu (t, &rcu_tasks_holdouts, rcu_tasks_holdout_list) -check_holdout_task (t); +check_holdout_task (t, needreport, &firstreport);rcu_read_unlock (); }
Signature · 1 line
1.8.1.5
PATCH v4 9/11 rcu: Improve RCU-tasks energy efficiency
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The current RCU-tasks implementation uses strict polling to detect callback arrivals. This works quite well, but is not so good for energy efficiency. This commit therefore replaces the strict polling with a wait queue.
--- kernel/rcu/update.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/rcu/update.c +12 -2
@@ -368,6 +368,7 @@ early_initcall(check_cpu_stall_init);/* Global list of callbacks and associated lock. */ static struct rcu_head *rcu_tasks_cbs_head;static struct rcu_head **rcu_tasks_cbs_tail = &rcu_tasks_cbs_head; +static DECLARE_WAIT_QUEUE_HEAD (rcu_tasks_cbs_wq);static DEFINE_RAW_SPINLOCK (rcu_tasks_cbs_lock);/* Track exiting tasks in order to allow them to be waited for. */
@@ -381,13 +382,17 @@ module_param(rcu_task_stall_timeout, int, 0644);void call_rcu_tasks (struct rcu_head *rhp,void (*func)(struct rcu_head *rhp)) {unsigned long flags; +bool needwake; rhp->next =NULL ; rhp->func = func;raw_spin_lock_irqsave (&rcu_tasks_cbs_lock, flags); + needwake = !rcu_tasks_cbs_head; *rcu_tasks_cbs_tail = rhp; rcu_tasks_cbs_tail = &rhp->next;raw_spin_unlock_irqrestore (&rcu_tasks_cbs_lock, flags); +if (needwake) +wake_up (&rcu_tasks_cbs_wq); }EXPORT_SYMBOL_GPL (call_rcu_tasks);
@@ -499,8 +504,12 @@ static int __noreturn rcu_tasks_kthread(void *arg)/* If there were none, wait a bit and start over. */ if (!list) { -schedule_timeout_interruptible (HZ ); -WARN_ON (signal_pending (current)); +wait_event_interruptible (rcu_tasks_cbs_wq, + rcu_tasks_cbs_head); +if (!rcu_tasks_cbs_head) { +WARN_ON (signal_pending (current)); +schedule_timeout_interruptible (HZ /10 ); + }continue ; }
@@ -606,6 +615,7 @@ static int __noreturn rcu_tasks_kthread(void *arg) list = next;cond_resched (); } +schedule_timeout_uninterruptible (HZ /10 ); } }
Signature · 1 line
1.8.1.5
PATCH v4 10/11 documentation: Add verbiage on RCU-tasks stall warning messages
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit documents RCU-tasks stall warning messages and also describes when to use the new cond_resched_rcu_qs() API.
--- Documentation/RCU/stallwarn.txt | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-)
Documentation/RCU/stallwarn.txt +24 -9
@@ -56,8 +56,20 @@ RCU_STALL_RAT_DELAY two jiffies. (This is a cpp macro, not a kernel configuration parameter.) -When a CPU detects that it is stalling, it will print a message similar -to the following: +rcupdate.rcu_task_stall_timeout + + This boot/sysfs parameter controls the RCU-tasks stall warning + interval. A value of zero or less suppresses RCU-tasks stall + warnings. A positive value sets the stall-warning interval + in jiffies. An RCU-tasks stall warning starts wtih the line: + + INFO: rcu_tasks detected stalls on tasks: + + And continues with the output of sched_show_task() for each + task stalling the current RCU-tasks grace period. + +For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling, +it will print a message similar to the following: INFO: rcu_sched_state detected stall on CPU 5 (t=2500 jiffies)
@@ -174,8 +186,12 @@ o A CPU looping with preemption disabled. This condition can o A CPU looping with bottom halves disabled. This condition can result in RCU-sched and RCU-bh stalls. -o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel - without invoking schedule(). +o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the + kernel without invoking schedule(). Note that cond_resched() + does not necessarily prevent RCU CPU stall warnings. Therefore, + if the looping in the kernel is really expected and desirable + behavior, you might need to replace some of the cond_resched() + calls with calls to cond_resched_rcu_qs(). o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might happen to preempt a low-priority task in the middle of an RCU
@@ -208,11 +224,10 @@ o A hardware failure. This is quite unlikely, but has occurred This resulted in a series of RCU CPU stall warnings, eventually leading the realization that the CPU had failed. -The RCU, RCU-sched, and RCU-bh implementations have CPU stall warning. -SRCU does not have its own CPU stall warnings, but its calls to -synchronize_sched() will result in RCU-sched detecting RCU-sched-related -CPU stalls. Please note that RCU only detects CPU stalls when there is -a grace period in progress. No grace period, no CPU stall warnings. +The RCU, RCU-sched, RCU-bh, and RCU-tasks implementations have CPU stall +warning. Note that SRCU does -not- have CPU stall warnings. Please note +that RCU only detects CPU stalls when there is a grace period in progress. +No grace period, no CPU stall warnings. To diagnose the cause of the stall, inspect the stack traces. The offending function will usually be near the top of the stack.
Signature · 1 line
1.8.1.5
PATCH v4 11/11 rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
It is expected that many sites will have CONFIG_TASKS_RCU=y, but will never actually invoke call_rcu_tasks(). For such sites, creating rcu_tasks_kthread() at boot is wasteful. This commit therefore defers creation of this kthread until the time of the first call_rcu_tasks().
This of course means that the first call_rcu_tasks() must be invoked from process context after the scheduler is fully operational.
--- kernel/rcu/update.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-)
kernel/rcu/update.c +26 -7
@@ -378,7 +378,12 @@ DEFINE_SRCU(tasks_rcu_exit_srcu);static int rcu_task_stall_timeout__read_mostly =HZ *60 *3 ;module_param (rcu_task_stall_timeout, int,0644 ); -/* Post an RCU-tasks callback. */ +static void rcu_spawn_tasks_kthread (void ); + +/* +* Post an RCU-tasks callback. First call must be from process context +* after the scheduler if fully operational. +*/ void call_rcu_tasks (struct rcu_head *rhp,void (*func)(struct rcu_head *rhp)) {unsigned long flags;
@@ -391,8 +396,10 @@ void call_rcu_tasks(struct rcu_head *rhp, void (*func)(struct rcu_head *rhp)) *rcu_tasks_cbs_tail = rhp; rcu_tasks_cbs_tail = &rhp->next;raw_spin_unlock_irqrestore (&rcu_tasks_cbs_lock, flags); -if (needwake) +if (needwake) { +rcu_spawn_tasks_kthread ();wake_up (&rcu_tasks_cbs_wq); + } }EXPORT_SYMBOL_GPL (call_rcu_tasks);
@@ -619,15 +626,27 @@ static int __noreturn rcu_tasks_kthread(void *arg) } } -/* Spawn rcu_tasks_kthread() at boot time. */ -static int __init rcu_spawn_tasks_kthread (void ) +/* Spawn rcu_tasks_kthread() at first call to call_rcu_tasks(). */ +static void rcu_spawn_tasks_kthread (void ) { -struct task_struct __maybe_unused *t; +static DEFINE_MUTEX (rcu_tasks_kthread_mutex); +static struct task_struct *rcu_tasks_kthread_ptr; +struct task_struct *t; +if (ACCESS_ONCE (rcu_tasks_kthread_ptr)) { +smp_mb ();/* Ensure caller sees full kthread. */ +return ; + } +mutex_lock (&rcu_tasks_kthread_mutex); +if (rcu_tasks_kthread_ptr) { +mutex_unlock (&rcu_tasks_kthread_mutex); +return ; + } t =kthread_run (rcu_tasks_kthread,NULL ,"rcu_tasks_kthread" );BUG_ON (IS_ERR (t)); -return 0 ; +smp_mb ();/* Ensure others see full kthread. */ +ACCESS_ONCE (rcu_tasks_kthread_ptr) = t; +mutex_unlock (&rcu_tasks_kthread_mutex); } -early_initcall (rcu_spawn_tasks_kthread);#endif /* #ifdef CONFIG_TASKS_RCU */
Signature · 1 line
1.8.1.5