Preview of fifth set of RCU changes for 3.3
Subsystems (4)
cover Preview of fifth set of RCU changes for 3.3
Hello!
This patchset adds to the earlier sets:
https://lkml.org/lkml/2011/11/2/363 https://lkml.org/lkml/2011/11/15/302 https://lkml.org/lkml/2011/11/28/588 https://lkml.org/lkml/2011/12/3/77
This fifth set adds rcutorture tests for the recently added srcu_read_lock_raw(), additional tracing to further evaluate RCU_FAST_NO_HZ, addition of an export to assist with rcutorture testing, bug fixes, and documentation. The patches are as follows:
1. Add rcutorture tests for srcu_read_lock_raw().
2. Add tracing to further validate RCU_FAST_NO_HZ by giving more
information on RCU's state at the time it decides to stop invoking callbacks.
3. Add the 2010 RCU API LWN article to the RCU documentation
(courtesy of Kees Cook).
4. Revert patch that permitted rtmutex acquisition with interrupts
disabled. Per LKML discussions (https://lkml.org/lkml/2011/12/5/63),
it was decided to illegalize IRQ-disable code sections that partially overlap with RCU read-side critical sections. Code to enforce this illegalization is in progress.
5. Add ACCESS_ONCE() to rcu_boost() return value.
6. Export cpu_up() so that rcutorture can exercise CPU hotplug
and RCU when rcutorture is built as a module.
For a testing-only version of this patchset from git, please see the following subject-to-rebase branch, based on 3.2-rc5:
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/next
Thanx, Paul
------------------------------------------------------------------------
b/Documentation/RCU/whatisRCU.txt | 1 + b/include/trace/events/rcu.h | 38 +++++++++++++++++++++++++++++--------- b/kernel/cpu.c | 1 + b/kernel/rcutiny.c | 10 ++++++++-- b/kernel/rcutiny_plugin.h | 25 +++++++++++++++++++++++++ b/kernel/rcutorture.c | 26 +++++++++++++++++++++++++- b/kernel/rcutree.c | 8 ++++++-- b/kernel/rcutree_plugin.h | 5 ----- b/kernel/rtmutex.c | 8 -------- kernel/rcutiny_plugin.h | 4 ++-- kernel/rcutree_plugin.h | 3 ++- 11 files changed, 99 insertions(+), 30 deletions(-)
RFC PATCH 1/6 rcu: Add rcutorture tests for srcu_read_lock_raw()
From: "Paul E. McKenney" <[email protected]>
This commit adds simple rcutorture tests for srcu_read_lock_raw() and srcu_read_unlock_raw(). It does not test doing srcu_read_lock_raw() in an exception handler and releasing it in the corresponding process context.
--- kernel/rcutorture.c | 26 +++++++++++++++++++++++++- 1 files changed, 25 insertions(+), 1 deletions(-)
kernel/rcutorture.c +25 -1
@@ -632,6 +632,30 @@ static struct rcu_torture_ops srcu_ops = { .name = "srcu" }; +static int srcu_torture_read_lock_raw(void) __acquires(&srcu_ctl) +{ +return srcu_read_lock_raw (&srcu_ctl); +} + +static void srcu_torture_read_unlock_raw(int idx) __releases(&srcu_ctl) +{ +srcu_read_unlock_raw (&srcu_ctl, idx); +} + +static struct rcu_torture_ops srcu_raw_ops = { + .init = srcu_torture_init, + .cleanup = srcu_torture_cleanup, + .readlock = srcu_torture_read_lock_raw, + .read_delay = srcu_read_delay, + .readunlock = srcu_torture_read_unlock_raw, + .completed = srcu_torture_completed, + .deferred_free = rcu_sync_torture_deferred_free, + .sync = srcu_torture_synchronize, + .cb_barrier =NULL , + .stats = srcu_torture_stats, + .name ="srcu_raw" +}; +static void srcu_torture_synchronize_expedited (void ) {synchronize_srcu_expedited (&srcu_ctl);
@@ -1591,7 +1615,7 @@ rcu_torture_init(void)static struct rcu_torture_ops *torture_ops[] = { &rcu_ops, &rcu_sync_ops, &rcu_expedited_ops, &rcu_bh_ops, &rcu_bh_sync_ops, &rcu_bh_expedited_ops, - &srcu_ops, &srcu_expedited_ops, + &srcu_ops, &srcu_raw_ops, &srcu_expedited_ops, &sched_ops, &sched_sync_ops, &sched_expedited_ops, };mutex_lock (&fullstop_mutex);
Signature · 1 line
1.7.8
RFC PATCH 2/6 rcu: Augment rcu_batch_end tracing for idle and callback state
From: "Paul E. McKenney" <[email protected]>
The current rcu_batch_end event trace records only the name of the RCU flavor and the total number of callbacks that remain queued on the current CPU. This is insufficient for testing and tuning the new dyntick-idle RCU_FAST_NO_HZ code, so this commit adds idle state along with whether or not any of the callbacks that were ready to invoke at the beginning of rcu_do_batch() are still queued.
--- include/trace/events/rcu.h | 38 +++++++++++++++++++++++++++++--------- kernel/rcutiny.c | 10 ++++++++-- kernel/rcutiny_plugin.h | 25 +++++++++++++++++++++++++ kernel/rcutree.c | 8 ++++++-- 4 files changed, 68 insertions(+), 13 deletions(-)
include/trace/events/rcu.h +29 -9
@@ -461,27 +461,46 @@ TRACE_EVENT(rcu_invoke_kfree_callback,/* * Tracepoint for exiting rcu_do_batch after RCU callbacks have been -* invoked. The first argument is the name of the RCU flavor and -* the second argument is number of callbacks actually invoked. +* invoked. The first argument is the name of the RCU flavor, +* the second argument is number of callbacks actually invoked, +* the third argument (cb) is whether or not any of the callbacks that +* were ready to invoke at the beginning of this batch are still +* queued, the fourth argument (nr) is the return value of need_resched(), +* the fifth argument (iit) is 1 if the current task is the idle task, +* and the sixth argument (risk) is the return value from +* rcu_is_callbacks_kthread(). */ TRACE_EVENT (rcu_batch_end, - TP_PROTO(char *rcuname, int callbacks_invoked), + TP_PROTO(char *rcuname, int callbacks_invoked, + bool cb, bool nr, bool iit, bool risk), -TP_ARGS (rcuname, callbacks_invoked), +TP_ARGS (rcuname, callbacks_invoked, cb, nr, iit, risk),TP_STRUCT__entry ( __field(char *, rcuname) __field(int, callbacks_invoked) + __field(bool, cb) + __field(bool, nr) + __field(bool, iit) +__field (bool, risk) ),TP_fast_assign ( __entry->rcuname = rcuname; __entry->callbacks_invoked = callbacks_invoked; - ), - -TP_printk ("%s CBs-invoked=%d" , - __entry->rcuname, __entry->callbacks_invoked) + __entry->cb = cb; + __entry->nr = nr; + __entry->iit = iit; + __entry->risk = risk; + ), + +TP_printk ("%s CBs-invoked=%d idle=%c%c%c%c" , + __entry->rcuname, __entry->callbacks_invoked, + __entry->cb ?'C' :'.' , + __entry->nr ?'S' :'.' , + __entry->iit ?'I' :'.' , + __entry->risk ?'R' :'.' ) );/*
@@ -524,7 +543,8 @@ TRACE_EVENT(rcu_torture_read,#define trace_rcu_batch_start (rcuname, qlen, blimit) do { } while (0)#define trace_rcu_invoke_callback (rcuname, rhp) do { } while (0)#define trace_rcu_invoke_kfree_callback (rcuname, rhp, offset) do { } while (0) -#define trace_rcu_batch_end (rcuname, callbacks_invoked) do { } while (0) +#define trace_rcu_batch_end (rcuname, callbacks_invoked, cb, nr, iit, risk) \ + do { } while (0)#define trace_rcu_torture_read (rcutorturename, rhp) do { } while (0)#endif /* #else #ifdef CONFIG_RCU_TRACE */
kernel/rcutiny.c +8 -2
@@ -259,7 +259,11 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)/* If no RCU callbacks ready to invoke, just return. */ if (&rcp->rcucblist == rcp->donetail) {RCU_TRACE (trace_rcu_batch_start (rcp->name,0 ,-1 )); -RCU_TRACE (trace_rcu_batch_end (rcp->name,0 )); +RCU_TRACE (trace_rcu_batch_end (rcp->name,0 , +ACCESS_ONCE (rcp->rcucblist), +need_resched (), +is_idle_task (current), +rcu_is_callbacks_kthread ()));return ; }
@@ -288,7 +292,9 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)RCU_TRACE (cb_count++); }RCU_TRACE (rcu_trace_sub_qlen (rcp, cb_count)); -RCU_TRACE (trace_rcu_batch_end (rcp->name, cb_count)); +RCU_TRACE (trace_rcu_batch_end (rcp->name, cb_count,0 ,need_resched (), +is_idle_task (current), +rcu_is_callbacks_kthread ())); }static void rcu_process_callbacks (struct softirq_action *unused)
kernel/rcutiny_plugin.h +25 -0
@@ -885,6 +885,19 @@ static void invoke_rcu_callbacks(void)wake_up (&rcu_kthread_wq); } +#ifdef CONFIG_RCU_TRACE + +/* +* Is the current CPU running the RCU-callbacks kthread? +* Caller must have preemption disabled. +*/ +static bool rcu_is_callbacks_kthread (void ) +{ +return rcu_kthread_task == current; +} + +#endif /* #ifdef CONFIG_RCU_TRACE */ +/* * This kthread invokes RCU callbacks whose grace periods have * elapsed. It is awakened as needed, and takes the place of the
@@ -938,6 +951,18 @@ void invoke_rcu_callbacks(void)raise_softirq (RCU_SOFTIRQ ); } +#ifdef CONFIG_RCU_TRACE + +/* +* There is no callback kthread, so this thread is never it. +*/ +static bool rcu_is_callbacks_kthread (void ) +{ +return false; +} + +#endif /* #ifdef CONFIG_RCU_TRACE */ +void rcu_init (void ) {open_softirq (RCU_SOFTIRQ , rcu_process_callbacks);
kernel/rcutree.c +6 -2
@@ -1373,7 +1373,9 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp)/* If no callbacks are ready, just return.*/ if (!cpu_has_callbacks_ready_to_invoke (rdp)) {trace_rcu_batch_start (rsp->name,0 ,0 ); -trace_rcu_batch_end (rsp->name,0 ); +trace_rcu_batch_end (rsp->name,0 , !!ACCESS_ONCE (rdp->nxtlist), +need_resched (),is_idle_task (current), +rcu_is_callbacks_kthread ());return ; }
@@ -1409,7 +1411,9 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) }local_irq_save (flags); -trace_rcu_batch_end (rsp->name, count); +trace_rcu_batch_end (rsp->name, count, !!list,need_resched (), +is_idle_task (current), +rcu_is_callbacks_kthread ());/* Update count, and requeue any remaining callbacks. */ rdp->qlen -= count;
Signature · 1 line
1.7.8
RFC PATCH 3/6 docs: Additional LWN links to RCU API
In reply to Paul E. McKenneyFrom: Kees Cook <[email protected]>
Tyler Hicks pointed me at an additional article on RCU and I figured it should probably be mentioned with the others.
--- Documentation/RCU/whatisRCU.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
Documentation/RCU/whatisRCU.txt +1 -0
@@ -4,6 +4,7 @@ to start learning about RCU: 1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/ 2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/ 3. RCU part 3: the RCU API http://lwn.net/Articles/264090/ +4. The RCU API, 2010 Edition http://lwn.net/Articles/418853/ What is RCU?
Signature · 1 line
1.7.8
RFC PATCH 4/6 Revert "rcu: Permit rt_mutex_unlock() with irqs disabled"
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This reverts commit 5342e269b2b58ee0b0b4168a94087faaa60d0567.
The approach taken in this patch was deemed too abusive to mutexes, and thus too likely to result in maintenance problems in the future. Instead, we will disallow RCU read-side critical sections that partially overlap with interrupt-disbled code segments.
--- kernel/rcutree_plugin.h | 5 ----- kernel/rtmutex.c | 8 -------- 2 files changed, 0 insertions(+), 13 deletions(-)
kernel/rcutree_plugin.h +0 -5
@@ -1165,8 +1165,6 @@ static void rcu_initiate_boost_trace(struct rcu_node *rnp)#endif /* #else #ifdef CONFIG_RCU_TRACE */ -static struct lock_class_key rcu_boost_class; -/* * Carry out RCU priority boosting on the task indicated by ->exp_tasks * or ->boost_tasks, advancing the pointer to the next task in the
@@ -1229,9 +1227,6 @@ static int rcu_boost(struct rcu_node *rnp)*/ t = container_of(tb, struct task_struct, rcu_node_entry);rt_mutex_init_proxy_locked (&mtx, t); -/* Avoid lockdep false positives. This rt_mutex is its own thing. */ -lockdep_set_class_and_name (&mtx.wait_lock, &rcu_boost_class, -"rcu_boost_mutex" ); t->rcu_boost_mutex = &mtx;raw_spin_unlock_irqrestore (&rnp->lock, flags);rt_mutex_lock (&mtx);/* Side effect: boosts task t's priority. */
kernel/rtmutex.c +0 -8
@@ -579,7 +579,6 @@ __rt_mutex_slowlock(struct rt_mutex *lock, int state, struct rt_mutex_waiter *waiter) {int ret =0 ; -int was_disabled;for (;;) {/* Try to acquire the lock: */
@@ -602,17 +601,10 @@ __rt_mutex_slowlock(struct rt_mutex *lock, int state,raw_spin_unlock (&lock->wait_lock); - was_disabled =irqs_disabled (); -if (was_disabled) -local_irq_enable (); -debug_rt_mutex_print_deadlock (waiter);schedule_rt_mutex (lock); -if (was_disabled) -local_irq_disable (); -raw_spin_lock (&lock->wait_lock);set_current_state (state); }
Signature · 1 line
1.7.8
RFC PATCH 5/6 rcu: Apply ACCESS_ONCE() to rcu_boost() return value
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Both TINY_RCU's and TREE_RCU's implementations of rcu_boost() access the ->boost_tasks and ->exp_tasks fields without preventing concurrent changes to these fields. This commit therefore applies ACCESS_ONCE in order to prevent compiler mischief.
--- kernel/rcutiny_plugin.h | 4 ++-- kernel/rcutree_plugin.h | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
kernel/rcutiny_plugin.h +2 -2
@@ -312,8 +312,8 @@ static int rcu_boost(void)rt_mutex_lock (&mtx);rt_mutex_unlock (&mtx);/* Keep lockdep happy. */ -return rcu_preempt_ctrlblk.boost_tasks !=NULL || - rcu_preempt_ctrlblk.exp_tasks !=NULL ; +return ACCESS_ONCE (rcu_preempt_ctrlblk.boost_tasks) !=NULL || +ACCESS_ONCE (rcu_preempt_ctrlblk.exp_tasks) !=NULL ; }/*
kernel/rcutree_plugin.h +2 -1
@@ -1232,7 +1232,8 @@ static int rcu_boost(struct rcu_node *rnp)rt_mutex_lock (&mtx);/* Side effect: boosts task t's priority. */ rt_mutex_unlock (&mtx);/* Keep lockdep happy. */ -return rnp->exp_tasks !=NULL || rnp->boost_tasks !=NULL ; +return ACCESS_ONCE (rnp->exp_tasks) !=NULL || +ACCESS_ONCE (rnp->boost_tasks) !=NULL ; }/*
Signature · 1 line
1.7.8
RFC PATCH 6/6 cpu: Export cpu_up()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Building rcutorture as a module requires cpu_up() as well as cpu_down() exported, so apply EXPORT_SYMBOL_GPL().
--- kernel/cpu.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/cpu.c +1 -0
@@ -380,6 +380,7 @@ out:cpu_maps_update_done ();return err; } +EXPORT_SYMBOL_GPL (cpu_up);#ifdef CONFIG_PM_SLEEP_SMP static cpumask_var_t frozen_cpus;
Signature · 1 line
1.7.8