RCU changes for v3.11
Subsystems (6)
Linus,
Please pull the latest RCU git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core-rcu-for-linus
HEAD: b1fe9987b78755719e8627d58409174ba00c24de Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu
The major changes:
- Simplify RCU's grace-period and callback processing based on the new numbering for callbacks.
- Removal of TINY_PREEMPT_RCU in favor of TREE_PREEMPT_RCU for single-CPU low-latency systems.
- SRCU-related changes and fixes.
- Miscellaneous fixes, including converting a few remaining printk()
calls to pr_*().
- Documentation updates.
Thanks,
Ingo
------------------>
Lai Jiangshan (1):
powerpc,kvm: fix imbalance srcu_read_[un]lock()
Paul E. McKenney (31):
rcu: Convert rcutree.c printk calls
rcu: Move code to apply callback-numbering simplifications
rcu: Make __note_new_gpnum() check for ends of prior grace periods
rcu: Rename note_new_gpnum() to note_gp_changes()
rcu: Convert rcutree_plugin.h printk calls
rcu: Switch callers from rcu_process_gp_end() to note_gp_changes()
rcu: Merge __rcu_process_gp_end() into __note_gp_changes()
rcu: Eliminate check_for_new_grace_period() wrapper function
rcu: Inline trivial wrapper function rcu_start_gp_per_cpu()
rcu: Move redundant call to note_gp_changes() into called function
nohz_full: Update based on Sedat Dilek review
nohz_full: Document additional restrictions
kthread: Add kworker kthreads to OS-jitter documentation
rcu: Remove "Experimental" flags
rcu: Drive quiescent-state-forcing delay from HZ
rcu: Merge adjacent identical ifdefs
rcu: Apply Dave Jones's NOCB Kconfig help feedback
rcu: Remove srcu_read_lock_raw() and srcu_read_unlock_raw().
rcu: Remove TINY_PREEMPT_RCU
rcu: Remove show_tiny_preempt_stats()
rcu: Remove rcu_preempt_check_callbacks()
rcu: Remove rcu_preempt_remove_callbacks()
rcu: Remove rcu_preempt_process_callbacks()
rcu: Simplify RCU_TINY RCU callback invocation
rcu: Remove check_cpu_stall_preempt()
rcu: Remove the CONFIG_TINY_RCU ifdefs in rcutiny.h
rcu: Remove rcu_preempt_note_context_switch()
rcu: Consolidate rcutiny_plugin.h ifdefs
rcu: Remove TINY_PREEMPT_RCU tracing documentation
rcu: Shrink TINY_RCU by moving exit_rcu()
rcu: Shrink TINY_RCU by reworking CPU-stall ifdefs
Documentation/RCU/checklist.txt | 6 - Documentation/RCU/torture.txt | 6 - Documentation/RCU/trace.txt | 100 +-- Documentation/RCU/whatisRCU.txt | 22 +- Documentation/kernel-per-CPU-kthreads.txt | 47 ++ Documentation/timers/NO_HZ.txt | 79 ++- arch/powerpc/kvm/book3s_hv.c | 2 +- include/linux/hardirq.h | 2 +- include/linux/rcupdate.h | 5 +- include/linux/rcutiny.h | 41 +- include/linux/rcutree.h | 3 +- include/linux/srcu.h | 43 -- init/Kconfig | 46 +- kernel/rcupdate.c | 29 +- kernel/rcutiny.c | 21 +- kernel/rcutiny_plugin.h | 1009 +---------------------------- kernel/rcutorture.c | 39 -- kernel/rcutree.c | 168 ++--- kernel/rcutree.h | 15 +- kernel/rcutree_plugin.h | 81 ++- 20 files changed, 316 insertions(+), 1448 deletions(-)
Documentation/RCU/checklist.txt +0 -6
@@ -354,12 +354,6 @@ over a rather long period of time, but improvements are always welcome! using RCU rather than SRCU, because RCU is almost always faster and easier to use than is SRCU. - If you need to enter your read-side critical section in a - hardirq or exception handler, and then exit that same read-side - critical section in the task that was interrupted, then you need - to srcu_read_lock_raw() and srcu_read_unlock_raw(), which avoid - the lockdep checking that would otherwise this practice illegal. - Also unlike other forms of RCU, explicit initialization and cleanup is required via init_srcu_struct() and cleanup_srcu_struct(). These are passed a "struct srcu_struct"
Documentation/RCU/torture.txt +0 -6
@@ -182,12 +182,6 @@ torture_type The type of RCU to test, with string values as follows: "srcu_expedited": srcu_read_lock(), srcu_read_unlock() and synchronize_srcu_expedited(). - "srcu_raw": srcu_read_lock_raw(), srcu_read_unlock_raw(), - and call_srcu(). - - "srcu_raw_sync": srcu_read_lock_raw(), srcu_read_unlock_raw(), - and synchronize_srcu(). - "sched": preempt_disable(), preempt_enable(), and call_rcu_sched().
Documentation/RCU/trace.txt +4 -96
117 lines not on this page · raw message
Documentation/RCU/whatisRCU.txt +7 -15
@@ -842,9 +842,7 @@ SRCU: Critical sections Grace period Barrier srcu_read_lock synchronize_srcu srcu_barrier srcu_read_unlock call_srcu - srcu_read_lock_raw synchronize_srcu_expedited - srcu_read_unlock_raw - srcu_dereference + srcu_dereference synchronize_srcu_expedited SRCU: Initialization/cleanup init_srcu_struct
@@ -865,38 +863,32 @@ list can be helpful: a. Will readers need to block? If so, you need SRCU. -b. Is it necessary to start a read-side critical section in a - hardirq handler or exception handler, and then to complete - this read-side critical section in the task that was - interrupted? If so, you need SRCU's srcu_read_lock_raw() and - srcu_read_unlock_raw() primitives. - -c. What about the -rt patchset? If readers would need to block +b. What about the -rt patchset? If readers would need to block in an non-rt kernel, you need SRCU. If readers would block in a -rt kernel, but not in a non-rt kernel, SRCU is not necessary. -d. Do you need to treat NMI handlers, hardirq handlers, +c. Do you need to treat NMI handlers, hardirq handlers, and code segments with preemption disabled (whether via preempt_disable(), local_irq_save(), local_bh_disable(), or some other mechanism) as if they were explicit RCU readers? If so, RCU-sched is the only choice that will work for you. -e. Do you need RCU grace periods to complete even in the face +d. Do you need RCU grace periods to complete even in the face of softirq monopolization of one or more of the CPUs? For example, is your code subject to network-based denial-of-service attacks? If so, you need RCU-bh. -f. Is your workload too update-intensive for normal use of +e. Is your workload too update-intensive for normal use of RCU, but inappropriate for other synchronization mechanisms? If so, consider SLAB_DESTROY_BY_RCU. But please be careful! -g. Do you need read-side critical sections that are respected +f. Do you need read-side critical sections that are respected even though they are in the middle of the idle loop, during user-mode execution, or on an offlined CPU? If so, SRCU is the only choice that will work for you. -h. Otherwise, use RCU. +g. Otherwise, use RCU. Of course, this all assumes that you have determined that RCU is in fact the right tool for your job.
Documentation/kernel-per-CPU-kthreads.txt +47 -0
53 lines not on this page · raw message
Documentation/timers/NO_HZ.txt +67 -12
110 lines not on this page · raw message
arch/powerpc/kvm/book3s_hv.c +1 -1
@@ -1864,7 +1864,7 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu) up_out:up_read (¤t->mm->mmap_sem); - goto out; + goto out_srcu; }int kvmppc_core_init_vm (struct kvm *kvm)
include/linux/hardirq.h +1 -1
@@ -128,7 +128,7 @@ extern void synchronize_irq(unsigned int irq);# define synchronize_irq (irq) barrier()#endif -#if defined(CONFIG_TINY_RCU ) || defined(CONFIG_TINY_PREEMPT_RCU ) +#if defined(CONFIG_TINY_RCU )static inline void rcu_nmi_enter (void ) {
include/linux/rcupdate.h +2 -3
@@ -216,6 +216,7 @@ static inline int rcu_preempt_depth(void)#endif /* #else #ifdef CONFIG_PREEMPT_RCU */ /* Internal to kernel */ +extern void rcu_init (void );extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern void rcu_check_callbacks (int cpu,int user);
@@ -239,8 +240,6 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev, struct task_struct *next) { }#endif /* CONFIG_RCU_USER_QS */ -extern void exit_rcu (void ); -/** * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers * @a: Code that RCU needs to pay attention to.
@@ -277,7 +276,7 @@ void wait_rcu_gp(call_rcu_func_t crf);#if defined(CONFIG_TREE_RCU ) || defined(CONFIG_TREE_PREEMPT_RCU )#include <linux/rcutree.h> -#elif defined(CONFIG_TINY_RCU ) || defined(CONFIG_TINY_PREEMPT_RCU ) +#elif defined(CONFIG_TINY_RCU )#include <linux/rcutiny.h> #else #error "Unknown RCU implementation specified to kernel configuration"
include/linux/rcutiny.h +4 -37
80 lines not on this page · raw message
include/linux/rcutree.h +2 -1
@@ -30,7 +30,6 @@#ifndef __LINUX_RCUTREE_H#define __LINUX_RCUTREE_H -extern void rcu_init (void );extern void rcu_note_context_switch (int cpu);extern int rcu_needs_cpu (int cpu,unsigned long *delta_jiffies);extern void rcu_cpu_stall_reset (void );
@@ -86,6 +85,8 @@ extern void rcu_force_quiescent_state(void);extern void rcu_bh_force_quiescent_state (void );extern void rcu_sched_force_quiescent_state (void ); +extern void exit_rcu (void ); +extern void rcu_scheduler_starting (void );extern int rcu_scheduler_active__read_mostly ;
include/linux/srcu.h +0 -43
47 lines not on this page · raw message
init/Kconfig +27 -19
86 lines not on this page · raw message
kernel/rcupdate.c +1 -28
@@ -104,31 +104,7 @@ void __rcu_read_unlock(void) }EXPORT_SYMBOL_GPL (__rcu_read_unlock); -/* -* Check for a task exiting while in a preemptible-RCU read-side -* critical section, clean up if so. No need to issue warnings, -* as debug_check_no_locks_held() already does this if lockdep -* is enabled. -*/ -void exit_rcu (void ) -{ -struct task_struct *t = current; - -if (likely (list_empty (¤t->rcu_node_entry))) -return ; - t->rcu_read_lock_nesting =1 ; -barrier (); - t->rcu_read_unlock_special =RCU_READ_UNLOCK_BLOCKED ; -__rcu_read_unlock (); -} - -#else /* #ifdef CONFIG_PREEMPT_RCU */ - -void exit_rcu (void ) -{ -} - -#endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_DEBUG_LOCK_ALLOC static struct lock_class_key rcu_lock_key;
@@ -145,9 +121,6 @@ static struct lock_class_key rcu_sched_lock_key;struct lockdep_map rcu_sched_lock_map =STATIC_LOCKDEP_MAP_INIT ("rcu_read_lock_sched" , &rcu_sched_lock_key);EXPORT_SYMBOL_GPL (rcu_sched_lock_map); -#endif - -#ifdef CONFIG_DEBUG_LOCK_ALLOC int debug_lockdep_rcu_enabled (void ) {
kernel/rcutiny.c +11 -10
@@ -44,7 +44,6 @@/* Forward declarations for rcutiny_plugin.h. */ struct rcu_ctrlblk ; -static void invoke_rcu_callbacks (void );static void __rcu_process_callbacks (struct rcu_ctrlblk *rcp);static void rcu_process_callbacks (struct softirq_action *unused); static void __call_rcu(struct rcu_head *head,
@@ -205,7 +204,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)*/ static int rcu_qsctr_help (struct rcu_ctrlblk *rcp) { -reset_cpu_stall_ticks (rcp); +RCU_TRACE (reset_cpu_stall_ticks (rcp));if (rcp->rcucblist !=NULL && rcp->donetail != rcp->curtail) { rcp->donetail = rcp->curtail;
@@ -227,7 +226,7 @@ void rcu_sched_qs(int cpu)local_irq_save (flags);if (rcu_qsctr_help (&rcu_sched_ctrlblk) +rcu_qsctr_help (&rcu_bh_ctrlblk)) -invoke_rcu_callbacks (); +raise_softirq (RCU_SOFTIRQ );local_irq_restore (flags); }
@@ -240,7 +239,7 @@ void rcu_bh_qs(int cpu)local_irq_save (flags);if (rcu_qsctr_help (&rcu_bh_ctrlblk)) -invoke_rcu_callbacks (); +raise_softirq (RCU_SOFTIRQ );local_irq_restore (flags); }
@@ -252,12 +251,11 @@ void rcu_bh_qs(int cpu)*/ void rcu_check_callbacks (int cpu,int user) { -check_cpu_stalls (); +RCU_TRACE (check_cpu_stalls ());if (user ||rcu_is_cpu_rrupt_from_idle ())rcu_sched_qs (cpu);else if (!in_softirq ())rcu_bh_qs (cpu); -rcu_preempt_check_callbacks (); }/*
@@ -278,7 +276,7 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)ACCESS_ONCE (rcp->rcucblist),need_resched (),is_idle_task (current), - rcu_is_callbacks_kthread())); + false));return ; }
@@ -290,7 +288,6 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp) *rcp->donetail =NULL ;if (rcp->curtail == rcp->donetail) rcp->curtail = &rcp->rcucblist; -rcu_preempt_remove_callbacks (rcp); rcp->donetail = &rcp->rcucblist;local_irq_restore (flags);
@@ -309,14 +306,13 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)RCU_TRACE (rcu_trace_sub_qlen (rcp, cb_count));RCU_TRACE (trace_rcu_batch_end (rcp->name, cb_count,0 ,need_resched (),is_idle_task (current), -rcu_is_callbacks_kthread ())); + false)); }static void rcu_process_callbacks (struct softirq_action *unused) {__rcu_process_callbacks (&rcu_sched_ctrlblk);__rcu_process_callbacks (&rcu_bh_ctrlblk); -rcu_preempt_process_callbacks (); }/*
@@ -382,3 +378,8 @@ void call_rcu_bh(struct rcu_head *head, void (*func)(struct rcu_head *rcu))__call_rcu (head, func, &rcu_bh_ctrlblk); }EXPORT_SYMBOL_GPL (call_rcu_bh); + +void rcu_init (void ) +{ +open_softirq (RCU_SOFTIRQ , rcu_process_callbacks); +}
kernel/rcutiny_plugin.h +35 -974
1,036 lines not on this page · raw message
kernel/rcutorture.c +0 -39
@@ -695,44 +695,6 @@ static struct rcu_torture_ops srcu_sync_ops = { .name = "srcu_sync" }; -static int srcu_torture_read_lock_raw(void) __acquires(&srcu_ctl) -{ -return srcu_read_lock_raw (&srcu_ctl); -} - -static void srcu_torture_read_unlock_raw(int idx) __releases(&srcu_ctl) -{ -srcu_read_unlock_raw (&srcu_ctl, idx); -} - -static struct rcu_torture_ops srcu_raw_ops = { - .init = rcu_sync_torture_init, - .readlock = srcu_torture_read_lock_raw, - .read_delay = srcu_read_delay, - .readunlock = srcu_torture_read_unlock_raw, - .completed = srcu_torture_completed, - .deferred_free = srcu_torture_deferred_free, - .sync = srcu_torture_synchronize, - .call =NULL , - .cb_barrier =NULL , - .stats = srcu_torture_stats, - .name ="srcu_raw" -}; - -static struct rcu_torture_ops srcu_raw_sync_ops = { - .init = rcu_sync_torture_init, - .readlock = srcu_torture_read_lock_raw, - .read_delay = srcu_read_delay, - .readunlock = srcu_torture_read_unlock_raw, - .completed = srcu_torture_completed, - .deferred_free = rcu_sync_torture_deferred_free, - .sync = srcu_torture_synchronize, - .call =NULL , - .cb_barrier =NULL , - .stats = srcu_torture_stats, - .name ="srcu_raw_sync" -}; -static void srcu_torture_synchronize_expedited (void ) {synchronize_srcu_expedited (&srcu_ctl);
@@ -1983,7 +1945,6 @@ rcu_torture_init(void) { &rcu_ops, &rcu_sync_ops, &rcu_expedited_ops, &rcu_bh_ops, &rcu_bh_sync_ops, &rcu_bh_expedited_ops, &srcu_ops, &srcu_sync_ops, &srcu_expedited_ops, - &srcu_raw_ops, &srcu_raw_sync_ops, &sched_ops, &sched_sync_ops, &sched_expedited_ops, };mutex_lock (&fullstop_mutex);
kernel/rcutree.c +44 -124
275 lines not on this page · raw message
kernel/rcutree.h +10 -5
@@ -343,12 +343,17 @@ struct rcu_data {#define RCU_FORCE_QS 3/* Need to force quiescent state. */ #define RCU_SIGNAL_INIT RCU_SAVE_DYNTICK -#define RCU_JIFFIES_TILL_FORCE_QS 3/* for rsp->jiffies_force_qs */ +#define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) +/* For jiffies_till_first_fqs and */ +/* and jiffies_till_next_fqs. */ -#define RCU_STALL_RAT_DELAY 2/* Allow other CPUs time */ -/* to take at least one */ -/* scheduling clock irq */ -/* before ratting on them. */ +#define RCU_JIFFIES_FQS_DIV 256/* Very large systems need more */ +/* delay between bouts of */ +/* quiescent-state forcing. */ + +#define RCU_STALL_RAT_DELAY 2/* Allow other CPUs time to take */ +/* at least one scheduling clock */ +/* irq before ratting on them. */ #define rcu_wait (cond) \ do { \
kernel/rcutree_plugin.h +53 -28
201 lines not on this page · raw message