rcu: resend of grace-period stall and cleanup patches
cover rcu: resend of grace-period stall and cleanup patches
Hello!
This patch series is a resend of the three RCU patches that are candidates for the upcoming 2.6.33 merge window, but that are not yet in -tip. These are:
1. A fix for a grace-period-stall bug that occurs on large
machines. This was partially fixed by commit 237c80c5, and this patch completes the fix using a much cleaner method. 237c80c5 would request another quiescent state for a CPU that just went offline and that might have already gone through a quiescent state for the current grace period. In contrast, this patch pushes the quiescent-state reporting up the rcu_node tree straightaway.
2. A cleanup removing function wrapping that has been unnecessary
since commit e74f4c45 caused CPUs going offline to give up their own callbacks.
3. A cleanup that gets rid of a few annoying #ifdefs by moving
the code from kernel/rcupdate.c and include/linux/rcupdate.h to include/linux/rcutiny.h, include/linux/rcutree.h, and kernel/rcutree.c.
Thanx, Paul
PATCH 1/3 rcu: fix grace-period-stall bug on large systems with CPU hotplug
When the last CPU of a given leaf rcu_node structure goes offline, all of the tasks queued on that leaf rcu_node structure (due to having blocked in their current RCU read-side critical sections) are requeued onto the root rcu_node structure. This requeuing is carried out by rcu_preempt_offline_tasks(). However, it is possible that these queued tasks are the only thing preventing the leaf rcu_node structure from reporting a quiescent state up the rcu_node hierarchy. Unfortunately, the old code would fail to do this reporting, resulting in a grace-period stall given the following sequence of events:
1. Kernel built for more than 32 CPUs on 32-bit systems or for more
than 64 CPUs on 64-bit systems, so that there is more than one rcu_node structure. (Or CONFIG_RCU_FANOUT is artificially set to a number smaller than CONFIG_NR_CPUS.)
2. The kernel is built with CONFIG_TREE_PREEMPT_RCU.
3. A task running on a CPU associated with a given leaf rcu_node
structure blocks while in an RCU read-side critical section -and- that CPU has not yet passed through a quiescent state for the current RCU grace period. This will cause the task to be queued on the leaf rcu_node's blocked_tasks[] array, in particular, on the element of this array corresponding to the current grace period.
4. Each of the remaining CPUs corresponding to this same leaf rcu_node
structure pass through a quiescent state. However, the task is still in its RCU read-side critical section, so these quiescent states cannot be reported further up the rcu_node hierarchy. Nevertheless, all bits in the leaf rcu_node structure's ->qsmask field are now zero.
5. Each of the remaining CPUs go offline. (The events in step
#4 and #5 can happen in any order as long as each CPU passes through a quiescent state before going offline.)
6. When the last CPU goes offline, __rcu_offline_cpu() will invoke
rcu_preempt_offline_tasks(), which will move the task to the root rcu_node structure, but without reporting a quiescent state up the rcu_node hierarchy (and this failure to report a quiescent state is the bug).
But because this leaf rcu_node structure's ->qsmask field is already zero and its ->block_tasks[] entries are all empty, force_quiescent_state() will skip this rcu_node structure.
Therefore, grace periods are now hung.
This patch abstracts some code out of rcu_read_unlock_special(), calling the result task_quiet() by analogy with cpu_quiet(), and invokes task_quiet() from both rcu_read_lock_special() and __rcu_offline_cpu(). Invoking task_quiet() from __rcu_offline_cpu() reports the quiescent state up the rcu_node hierarchy, fixing the bug. This ends up requiring a separate lock_class_key per level of the rcu_node hierarchy, which this patch also provides.
--- kernel/rcutree.c | 40 +++++++++++---------- kernel/rcutree.h | 3 ++ kernel/rcutree_plugin.h | 85 +++++++++++++++++++++++++++++++++------------- 3 files changed, 85 insertions(+), 43 deletions(-)
kernel/rcutree.c +21 -19
@@ -51,7 +51,7 @@/* Data structures. */ -static struct lock_class_key rcu_root_class; +static struct lock_class_key rcu_node_class[NUM_RCU_LVLS ];#define RCU_STATE_INITIALIZER (name) { \ .level = { &name.node[0] }, \
@@ -936,6 +936,7 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp) {unsigned long flags;unsigned long mask; +int need_quiet =0 ;struct rcu_data *rdp = rsp->rda[cpu];struct rcu_node *rnp;
@@ -949,29 +950,30 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp)spin_lock (&rnp->lock);/* irqs already disabled. */ rnp->qsmaskinit &= ~mask;if (rnp->qsmaskinit !=0 ) { -spin_unlock (&rnp->lock);/* irqs remain disabled. */ +if (rnp != rdp->mynode) +spin_unlock (&rnp->lock);/* irqs remain disabled. */ break ; } - -/* -* If there was a task blocking the current grace period, -* and if all CPUs have checked in, we need to propagate -* the quiescent state up the rcu_node hierarchy. But that -* is inconvenient at the moment due to deadlock issues if -* this should end the current grace period. So set the -* offlined CPU's bit in ->qsmask in order to force the -* next force_quiescent_state() invocation to clean up this -* mess in a deadlock-free manner. -*/ -if (rcu_preempt_offline_tasks (rsp, rnp, rdp) && !rnp->qsmask) - rnp->qsmask |= mask; - +if (rnp == rdp->mynode) + need_quiet =rcu_preempt_offline_tasks (rsp, rnp, rdp); +else +spin_unlock (&rnp->lock);/* irqs remain disabled. */ mask = rnp->grpmask; -spin_unlock (&rnp->lock);/* irqs remain disabled. */ rnp = rnp->parent; }while (rnp !=NULL ); -spin_unlock_irqrestore (&rsp->onofflock, flags); +/* +* We still hold the leaf rcu_node structure lock here, and +* irqs are still disabled. The reason for this subterfuge is +* because invoking task_quiet() with ->onofflock held leads +* to deadlock. +*/ +spin_unlock (&rsp->onofflock);/* irqs remain disabled. */ + rnp = rdp->mynode; +if (need_quiet) +task_quiet (rnp, flags); +else +spin_unlock_irqrestore (&rnp->lock, flags);rcu_adopt_orphan_cbs (rsp); }
@@ -1731,6 +1733,7 @@ static void __init rcu_init_one(struct rcu_state *rsp) rnp = rsp->level[i];for (j =0 ; j < rsp->levelcnt[i]; j++, rnp++) {spin_lock_init (&rnp->lock); +lockdep_set_class (&rnp->lock, &rcu_node_class[i]); rnp->gpnum =0 ; rnp->qsmask =0 ; rnp->qsmaskinit =0 ;
@@ -1753,7 +1756,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)INIT_LIST_HEAD (&rnp->blocked_tasks[1 ]); } } -lockdep_set_class (&rcu_get_root (rsp)->lock, &rcu_root_class); }/*
kernel/rcutree.h +3 -0
@@ -305,6 +305,9 @@ static void rcu_bootup_announce(void);long rcu_batches_completed (void );static void rcu_preempt_note_context_switch (int cpu);static int rcu_preempted_readers (struct rcu_node *rnp); +#ifdef CONFIG_HOTPLUG_CPU +static void task_quiet (struct rcu_node *rnp,unsigned long flags); +#endif /* #ifdef CONFIG_HOTPLUG_CPU */ #ifdef CONFIG_RCU_CPU_STALL_DETECTOR static void rcu_print_task_stall (struct rcu_node *rnp);#endif /* #ifdef CONFIG_RCU_CPU_STALL_DETECTOR */
kernel/rcutree_plugin.h +61 -24
134 lines not on this page · raw message
Signature · 1 line
1.5.2.5
PATCH 2/3 rcu: eliminate unneeded function wrapping
In reply to Paul E. McKenneyFrom: Paul E. McKenney <[email protected]>
The functions rcu_init() is a wrapper for __rcu_init(), and also sets up the CPU-hotplug notifier for rcu_barrier_cpu_hotplug(). But TINY_RCU doesn't need CPU-hotplug notification, and the rcu_barrier_cpu_hotplug() is a simple wrapper for rcu_cpu_notify(). So push rcu_init() out to kernel/rcutree.c and kernel/rcutiny.c and get rid of the wrapper function rcu_barrier_cpu_hotplug().
--- include/linux/rcutiny.h | 2 -- include/linux/rcutree.h | 3 --- kernel/rcupdate.c | 22 ---------------------- kernel/rcutiny.c | 11 +---------- kernel/rcutree.c | 17 ++++++++++++++--- 5 files changed, 15 insertions(+), 40 deletions(-)
include/linux/rcutiny.h +0 -2
@@ -38,7 +38,6 @@ void rcu_bh_qs(int cpu);#define rcu_init_sched () do { } while (0)extern void rcu_check_callbacks (int cpu,int user); -extern void __rcu_init (void );/* * Return the number of grace periods.
@@ -69,7 +68,6 @@ static inline void synchronize_rcu_bh_expedited(void) }struct notifier_block ; -extern int rcu_cpu_notify (struct notifier_block *self,unsigned long action,void *hcpu);#ifdef CONFIG_NO_HZ
include/linux/rcutree.h +0 -3
@@ -34,8 +34,6 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu); -extern int rcu_cpu_notify (struct notifier_block *self, -unsigned long action,void *hcpu);extern int rcu_needs_cpu (int cpu);extern int rcu_expedited_torture_stats (char *page);
@@ -83,7 +81,6 @@ static inline void synchronize_rcu_bh_expedited(void)synchronize_sched_expedited (); } -extern void __rcu_init (void );extern void rcu_check_callbacks (int cpu,int user);extern long rcu_batches_completed (void );
kernel/rcupdate.c +0 -22
@@ -161,28 +161,6 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_bh);#endif /* #ifndef CONFIG_TINY_RCU */ -static int __cpuinit rcu_barrier_cpu_hotplug (struct notifier_block *self, -unsigned long action,void *hcpu) -{ -return rcu_cpu_notify (self, action, hcpu); -} - -void __init rcu_init (void ) -{ -int i; - -__rcu_init (); -cpu_notifier (rcu_barrier_cpu_hotplug,0 ); - -/* -* We don't need protection against CPU-hotplug here because -* this is called early in boot, before either interrupts -* or the scheduler are operational. -*/ -for_each_online_cpu (i) -rcu_barrier_cpu_hotplug (NULL ,CPU_UP_PREPARE , (void *)(long )i); -} -void rcu_scheduler_starting (void ) {WARN_ON (num_online_cpus () !=1 );
kernel/rcutiny.c +1 -10
@@ -178,15 +178,6 @@ static void rcu_process_callbacks(struct softirq_action *unused) }/* -* Null function to handle CPU being onlined. Longer term, we want to -* make TINY_RCU avoid using rcupdate.c, but later... -*/ -int rcu_cpu_notify (struct notifier_block *self,unsigned long action,void *hcpu) -{ -return NOTIFY_OK ; -} - -/* * Wait for a grace period to elapse. But it is illegal to invoke * synchronize_sched() from within an RCU read-side critical section. * Therefore, any legal call to synchronize_sched() is a quiescent
@@ -285,7 +276,7 @@ void rcu_barrier_sched(void) }EXPORT_SYMBOL_GPL (rcu_barrier_sched); -void __rcu_init (void ) +void __init rcu_init (void ) {open_softirq (RCU_SOFTIRQ , rcu_process_callbacks); }
kernel/rcutree.c +14 -3
@@ -1644,8 +1644,8 @@ static void __cpuinit rcu_online_cpu(int cpu)/* * Handle CPU online/offline notification events. */ -int __cpuinit rcu_cpu_notify (struct notifier_block *self, -unsigned long action,void *hcpu) +static int __cpuinit rcu_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) {long cpu = (long )hcpu;
@@ -1781,8 +1781,10 @@ do { \ } \ }while (0 ) -void __init __rcu_init(void) +void __init rcu_init(void) { +int i; +rcu_bootup_announce ();#ifdef CONFIG_RCU_CPU_STALL_DETECTOR printk (KERN_INFO "RCU-based detection of stalled CPUs is enabled.\n" );
@@ -1791,6 +1793,15 @@ void __init __rcu_init(void)RCU_INIT_FLAVOR (&rcu_bh_state, rcu_bh_data);__rcu_init_preempt ();open_softirq (RCU_SOFTIRQ , rcu_process_callbacks); + +/* +* We don't need protection against CPU-hotplug here because +* this is called early in boot, before either interrupts +* or the scheduler are operational. +*/ +cpu_notifier (rcu_cpu_notify,0 ); +for_each_online_cpu (i) +rcu_cpu_notify (NULL ,CPU_UP_PREPARE , (void *)(long )i); }#include "rcutree_plugin.h"
Signature · 1 line
1.5.2.5
PATCH 3/3 rcu: re-arrange code to reduce #ifdef pain
In reply to Paul E. McKenneyFrom: Paul E. McKenney <[email protected]>
Remove #ifdefs from kernel/rcupdate.c and include/linux/rcupdate.h by moving code to include/linux/rcutiny.h, include/linux/rcutree.h, and kernel/rcutree.c. Also remove some definitions that are no longer used.
--- include/linux/rcupdate.h | 12 ----- include/linux/rcutiny.h | 11 +++++ include/linux/rcutree.h | 4 +- kernel/rcupdate.c | 104 ---------------------------------------------- kernel/rcutree.c | 80 +++++++++++++++++++++++++++++++++++ kernel/rcutree_plugin.h | 24 +++++++++++ 6 files changed, 118 insertions(+), 117 deletions(-)
include/linux/rcupdate.h +0 -12
@@ -52,11 +52,6 @@ struct rcu_head { };/* Exported common interfaces */ -#ifdef CONFIG_TREE_PREEMPT_RCU -extern void synchronize_rcu (void ); -#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */ -#define synchronize_rcu synchronize_sched -#endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */ extern void synchronize_rcu_bh (void );extern void synchronize_sched (void );extern void rcu_barrier (void );
@@ -67,13 +62,6 @@ extern int sched_expedited_torture_stats(char *page);/* Internal to kernel */ extern void rcu_init (void ); -extern void rcu_scheduler_starting (void ); -#ifndef CONFIG_TINY_RCU -extern int rcu_needs_cpu (int cpu); -#else -static inline int rcu_needs_cpu (int cpu) {return 0 ; } -#endif -extern int rcu_scheduler_active;#if defined(CONFIG_TREE_RCU ) || defined(CONFIG_TREE_PREEMPT_RCU )#include <linux/rcutree.h>
include/linux/rcutiny.h +11 -0
@@ -39,6 +39,11 @@ void rcu_bh_qs(int cpu);#define rcu_init_sched () do { } while (0)extern void rcu_check_callbacks (int cpu,int user); +static inline int rcu_needs_cpu (int cpu) +{ +return 0 ; +} +/* * Return the number of grace periods. */
@@ -57,6 +62,8 @@ static inline long rcu_batches_completed_bh(void)extern int rcu_expedited_torture_stats (char *page); +#define synchronize_rcu synchronize_sched +static inline void synchronize_rcu_expedited (void ) {synchronize_sched ();
@@ -86,6 +93,10 @@ static inline void rcu_exit_nohz(void)#endif /* #else #ifdef CONFIG_NO_HZ */ +static inline void rcu_scheduler_starting (void ) +{ +} +static inline void exit_rcu (void ) { }
include/linux/rcutree.h +3 -1
@@ -35,12 +35,14 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu); +extern void rcu_scheduler_starting (void );extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU extern void __rcu_read_lock (void );extern void __rcu_read_unlock (void ); +extern void synchronize_rcu (void );extern void exit_rcu (void );#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
@@ -55,7 +57,7 @@ static inline void __rcu_read_unlock(void)preempt_enable (); } -#define __synchronize_sched () synchronize_rcu() +#define synchronize_rcu synchronize_schedstatic inline void exit_rcu (void ) {
kernel/rcupdate.c +0 -104
119 lines not on this page · raw message
kernel/rcutree.c +80 -0
104 lines not on this page · raw message
kernel/rcutree_plugin.h +24 -0
@@ -425,6 +425,30 @@ void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) }EXPORT_SYMBOL_GPL (call_rcu); +/** +* synchronize_rcu - wait until a grace period has elapsed. +* +* Control will return to the caller some time after a full grace +* period has elapsed, in other words after all currently executing RCU +* read-side critical sections have completed. RCU read-side critical +* sections are delimited by rcu_read_lock() and rcu_read_unlock(), +* and may be nested. +*/ +void synchronize_rcu (void ) +{ +struct rcu_synchronize rcu; + +if (!rcu_scheduler_active) +return ; + +init_completion (&rcu.completion); +/* Will wake me after RCU finished. */ +call_rcu (&rcu.head, wakeme_after_rcu); +/* Wait for it. */ +wait_for_completion (&rcu.completion); +} +EXPORT_SYMBOL_GPL (synchronize_rcu); +/* * Wait for an rcu-preempt grace period. We are supposed to expedite the * grace period, but this is the crude slow compatability hack, so just
Signature · 1 line
1.5.2.5
Quoted 8 lines · * Paul E. McKenney ([email protected]) wrote:
Hello!
This patch series is a resend of the three RCU patches that are candidates for the upcoming 2.6.33 merge window, but that are not yet in -tip. These are:
1. A fix for a grace-period-stall bug that occurs on large
machines.
[...]
Hi Paul,
I was thinking about the last bugs you discovered. Some caracteristics they had in common were that they occur only on large marchines (32+ or 64+ CPUs). This is caused by the fact that some of your code is only covered by tests when the number of CPUs go over the architecture size (in bits).
I managed to cover this kind of scenario with smaller state-space in the LTTng formal models (but it also applies to kernel code) by tweaking the code, with bitmasks, to ensure that the number of bits the code uses is, e.g., no more than the minimum amount of required bits. Therefore, you are ensured to run into overflow scenarios either more quickly or, as in this case, on decently-sized hardware.
Hope this helps,
Thanks,
Mathieu
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 19 lines · Mathieu Desnoyers
Quoted 8 lines · * Paul E. McKenney ([email protected]) wrote:
Hello!
This patch series is a resend of the three RCU patches that are candidates for the upcoming 2.6.33 merge window, but that are not yet in -tip. These are:
1. A fix for a grace-period-stall bug that occurs on large
machines.[...]
Hi Paul,
I was thinking about the last bugs you discovered. Some caracteristics they had in common were that they occur only on large marchines (32+ or 64+ CPUs). This is caused by the fact that some of your code is only covered by tests when the number of CPUs go over the architecture size (in bits).
Quoted 6 lines
I managed to cover this kind of scenario with smaller state-space in the LTTng formal models (but it also applies to kernel code) by tweaking the code, with bitmasks, to ensure that the number of bits the code uses is, e.g., no more than the minimum amount of required bits. Therefore, you are ensured to run into overflow scenarios either more quickly or, as in this case, on decently-sized hardware.
You mean by setting CONFIG_RCU_FANOUT=2 in order to get three levels of rcu_node hierarchy on an eight-CPU machine, which would otherwise require more than 1024 CPU on a 32-bit system or more that 4096 CPUs on a 64-bit system? ;-)
http://paulmck.livejournal.com/14969.html
But yes, the largest machine I have access to has "only" 128 CPUs, and it is often heavily used by others. So I heartily agree with your point, which is that we should use various techniques to test code on smaller machines in ways that larger machines will stress it. Of course, my favorite such technique is differential profiling, which allows performance results collected on small machines to reveal problems that would only show up on large machines:
http://www.rdrop.com/users/paulmck/scalability/paper/profiling.2002.06.04.pdf
(This is a revision of a paper that appeared in the 1995 MASCOTS conference and in the 1999 Software Practice & Experience journal.)
Thanx, Paul
[tip:core/rcu] rcu: Fix grace-period-stall bug on large systems with CPU hotplug
In reply to Paul E. McKenneyCommit-ID: b668c9cf3e58739dac54a1d6f42f2b4bdd980b3e Gitweb: http://git.kernel.org/tip/b668c9cf3e58739dac54a1d6f42f2b4bdd980b3e Author: Paul E. McKenney <[email protected]> AuthorDate: Sun, 22 Nov 2009 08:53:48 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 22 Nov 2009 18:58:15 +0100
rcu: Fix grace-period-stall bug on large systems with CPU hotplug
When the last CPU of a given leaf rcu_node structure goes offline, all of the tasks queued on that leaf rcu_node structure (due to having blocked in their current RCU read-side critical sections) are requeued onto the root rcu_node structure. This requeuing is carried out by rcu_preempt_offline_tasks(). However, it is possible that these queued tasks are the only thing preventing the leaf rcu_node structure from reporting a quiescent state up the rcu_node hierarchy. Unfortunately, the old code would fail to do this reporting, resulting in a grace-period stall given the following sequence of events:
1. Kernel built for more than 32 CPUs on 32-bit systems or for more
than 64 CPUs on 64-bit systems, so that there is more than one rcu_node structure. (Or CONFIG_RCU_FANOUT is artificially set to a number smaller than CONFIG_NR_CPUS.)
2. The kernel is built with CONFIG_TREE_PREEMPT_RCU.
3. A task running on a CPU associated with a given leaf rcu_node
structure blocks while in an RCU read-side critical section -and- that CPU has not yet passed through a quiescent state for the current RCU grace period. This will cause the task to be queued on the leaf rcu_node's blocked_tasks[] array, in particular, on the element of this array corresponding to the current grace period.
4. Each of the remaining CPUs corresponding to this same leaf rcu_node
structure pass through a quiescent state. However, the task is still in its RCU read-side critical section, so these quiescent states cannot be reported further up the rcu_node hierarchy. Nevertheless, all bits in the leaf rcu_node structure's ->qsmask field are now zero.
5. Each of the remaining CPUs go offline. (The events in step
#4 and #5 can happen in any order as long as each CPU passes through a quiescent state before going offline.)
6. When the last CPU goes offline, __rcu_offline_cpu() will invoke
rcu_preempt_offline_tasks(), which will move the task to the root rcu_node structure, but without reporting a quiescent state up the rcu_node hierarchy (and this failure to report a quiescent state is the bug).
But because this leaf rcu_node structure's ->qsmask field is already zero and its ->block_tasks[] entries are all empty, force_quiescent_state() will skip this rcu_node structure.
Therefore, grace periods are now hung.
This patch abstracts some code out of rcu_read_unlock_special(), calling the result task_quiet() by analogy with cpu_quiet(), and invokes task_quiet() from both rcu_read_lock_special() and __rcu_offline_cpu(). Invoking task_quiet() from __rcu_offline_cpu() reports the quiescent state up the rcu_node hierarchy, fixing the bug. This ends up requiring a separate lock_class_key per level of the rcu_node hierarchy, which this patch also provides.
LKML-Reference: <12589088301770-git-send-email->
--- kernel/rcutree.c | 40 +++++++++++---------- kernel/rcutree.h | 3 ++ kernel/rcutree_plugin.h | 85 +++++++++++++++++++++++++++++++++------------- 3 files changed, 85 insertions(+), 43 deletions(-)
kernel/rcutree.c +21 -19
@@ -51,7 +51,7 @@/* Data structures. */ -static struct lock_class_key rcu_root_class; +static struct lock_class_key rcu_node_class[NUM_RCU_LVLS ];#define RCU_STATE_INITIALIZER (name) { \ .level = { &name.node[0] }, \
@@ -936,6 +936,7 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp) {unsigned long flags;unsigned long mask; +int need_quiet =0 ;struct rcu_data *rdp = rsp->rda[cpu];struct rcu_node *rnp;
@@ -949,29 +950,30 @@ static void __rcu_offline_cpu(int cpu, struct rcu_state *rsp)spin_lock (&rnp->lock);/* irqs already disabled. */ rnp->qsmaskinit &= ~mask;if (rnp->qsmaskinit !=0 ) { -spin_unlock (&rnp->lock);/* irqs remain disabled. */ +if (rnp != rdp->mynode) +spin_unlock (&rnp->lock);/* irqs remain disabled. */ break ; } - -/* -* If there was a task blocking the current grace period, -* and if all CPUs have checked in, we need to propagate -* the quiescent state up the rcu_node hierarchy. But that -* is inconvenient at the moment due to deadlock issues if -* this should end the current grace period. So set the -* offlined CPU's bit in ->qsmask in order to force the -* next force_quiescent_state() invocation to clean up this -* mess in a deadlock-free manner. -*/ -if (rcu_preempt_offline_tasks (rsp, rnp, rdp) && !rnp->qsmask) - rnp->qsmask |= mask; - +if (rnp == rdp->mynode) + need_quiet =rcu_preempt_offline_tasks (rsp, rnp, rdp); +else +spin_unlock (&rnp->lock);/* irqs remain disabled. */ mask = rnp->grpmask; -spin_unlock (&rnp->lock);/* irqs remain disabled. */ rnp = rnp->parent; }while (rnp !=NULL ); -spin_unlock_irqrestore (&rsp->onofflock, flags); +/* +* We still hold the leaf rcu_node structure lock here, and +* irqs are still disabled. The reason for this subterfuge is +* because invoking task_quiet() with ->onofflock held leads +* to deadlock. +*/ +spin_unlock (&rsp->onofflock);/* irqs remain disabled. */ + rnp = rdp->mynode; +if (need_quiet) +task_quiet (rnp, flags); +else +spin_unlock_irqrestore (&rnp->lock, flags);rcu_adopt_orphan_cbs (rsp); }
@@ -1731,6 +1733,7 @@ static void __init rcu_init_one(struct rcu_state *rsp) rnp = rsp->level[i];for (j =0 ; j < rsp->levelcnt[i]; j++, rnp++) {spin_lock_init (&rnp->lock); +lockdep_set_class (&rnp->lock, &rcu_node_class[i]); rnp->gpnum =0 ; rnp->qsmask =0 ; rnp->qsmaskinit =0 ;
@@ -1753,7 +1756,6 @@ static void __init rcu_init_one(struct rcu_state *rsp)INIT_LIST_HEAD (&rnp->blocked_tasks[1 ]); } } -lockdep_set_class (&rcu_get_root (rsp)->lock, &rcu_root_class); }/*
kernel/rcutree.h +3 -0
@@ -305,6 +305,9 @@ static void rcu_bootup_announce(void);long rcu_batches_completed (void );static void rcu_preempt_note_context_switch (int cpu);static int rcu_preempted_readers (struct rcu_node *rnp); +#ifdef CONFIG_HOTPLUG_CPU +static void task_quiet (struct rcu_node *rnp,unsigned long flags); +#endif /* #ifdef CONFIG_HOTPLUG_CPU */ #ifdef CONFIG_RCU_CPU_STALL_DETECTOR static void rcu_print_task_stall (struct rcu_node *rnp);#endif /* #ifdef CONFIG_RCU_CPU_STALL_DETECTOR */
kernel/rcutree_plugin.h +61 -24
134 lines not on this page · raw message
[tip:core/rcu] rcu: Eliminate unneeded function wrapping
In reply to Paul E. McKenneyCommit-ID: 9f680ab41485edfdc96331b70afa7513aa0a7720 Gitweb: http://git.kernel.org/tip/9f680ab41485edfdc96331b70afa7513aa0a7720 Author: Paul E. McKenney <[email protected]> AuthorDate: Sun, 22 Nov 2009 08:53:49 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 22 Nov 2009 18:58:16 +0100
rcu: Eliminate unneeded function wrapping
The functions rcu_init() is a wrapper for __rcu_init(), and also sets up the CPU-hotplug notifier for rcu_barrier_cpu_hotplug(). But TINY_RCU doesn't need CPU-hotplug notification, and the rcu_barrier_cpu_hotplug() is a simple wrapper for rcu_cpu_notify().
So push rcu_init() out to kernel/rcutree.c and kernel/rcutiny.c and get rid of the wrapper function rcu_barrier_cpu_hotplug().
LKML-Reference: <12589088302320-git-send-email->
--- include/linux/rcutiny.h | 2 -- include/linux/rcutree.h | 3 --- kernel/rcupdate.c | 22 ---------------------- kernel/rcutiny.c | 11 +---------- kernel/rcutree.c | 17 ++++++++++++++--- 5 files changed, 15 insertions(+), 40 deletions(-)
include/linux/rcutiny.h +0 -2
@@ -38,7 +38,6 @@ void rcu_bh_qs(int cpu);#define rcu_init_sched () do { } while (0)extern void rcu_check_callbacks (int cpu,int user); -extern void __rcu_init (void );/* * Return the number of grace periods.
@@ -69,7 +68,6 @@ static inline void synchronize_rcu_bh_expedited(void) }struct notifier_block ; -extern int rcu_cpu_notify (struct notifier_block *self,unsigned long action,void *hcpu);#ifdef CONFIG_NO_HZ
include/linux/rcutree.h +0 -3
@@ -34,8 +34,6 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu); -extern int rcu_cpu_notify (struct notifier_block *self, -unsigned long action,void *hcpu);extern int rcu_needs_cpu (int cpu);extern int rcu_expedited_torture_stats (char *page);
@@ -83,7 +81,6 @@ static inline void synchronize_rcu_bh_expedited(void)synchronize_sched_expedited (); } -extern void __rcu_init (void );extern void rcu_check_callbacks (int cpu,int user);extern long rcu_batches_completed (void );
kernel/rcupdate.c +0 -22
@@ -161,28 +161,6 @@ EXPORT_SYMBOL_GPL(synchronize_rcu_bh);#endif /* #ifndef CONFIG_TINY_RCU */ -static int __cpuinit rcu_barrier_cpu_hotplug (struct notifier_block *self, -unsigned long action,void *hcpu) -{ -return rcu_cpu_notify (self, action, hcpu); -} - -void __init rcu_init (void ) -{ -int i; - -__rcu_init (); -cpu_notifier (rcu_barrier_cpu_hotplug,0 ); - -/* -* We don't need protection against CPU-hotplug here because -* this is called early in boot, before either interrupts -* or the scheduler are operational. -*/ -for_each_online_cpu (i) -rcu_barrier_cpu_hotplug (NULL ,CPU_UP_PREPARE , (void *)(long )i); -} -void rcu_scheduler_starting (void ) {WARN_ON (num_online_cpus () !=1 );
kernel/rcutiny.c +1 -10
@@ -178,15 +178,6 @@ static void rcu_process_callbacks(struct softirq_action *unused) }/* -* Null function to handle CPU being onlined. Longer term, we want to -* make TINY_RCU avoid using rcupdate.c, but later... -*/ -int rcu_cpu_notify (struct notifier_block *self,unsigned long action,void *hcpu) -{ -return NOTIFY_OK ; -} - -/* * Wait for a grace period to elapse. But it is illegal to invoke * synchronize_sched() from within an RCU read-side critical section. * Therefore, any legal call to synchronize_sched() is a quiescent
@@ -285,7 +276,7 @@ void rcu_barrier_sched(void) }EXPORT_SYMBOL_GPL (rcu_barrier_sched); -void __rcu_init (void ) +void __init rcu_init (void ) {open_softirq (RCU_SOFTIRQ , rcu_process_callbacks); }
kernel/rcutree.c +14 -3
@@ -1644,8 +1644,8 @@ static void __cpuinit rcu_online_cpu(int cpu)/* * Handle CPU online/offline notification events. */ -int __cpuinit rcu_cpu_notify (struct notifier_block *self, -unsigned long action,void *hcpu) +static int __cpuinit rcu_cpu_notify (struct notifier_block *self, +unsigned long action,void *hcpu) {long cpu = (long )hcpu;
@@ -1781,8 +1781,10 @@ do { \ } \ }while (0 ) -void __init __rcu_init(void) +void __init rcu_init(void) { +int i; +rcu_bootup_announce ();#ifdef CONFIG_RCU_CPU_STALL_DETECTOR printk (KERN_INFO "RCU-based detection of stalled CPUs is enabled.\n" );
@@ -1791,6 +1793,15 @@ void __init __rcu_init(void)RCU_INIT_FLAVOR (&rcu_bh_state, rcu_bh_data);__rcu_init_preempt ();open_softirq (RCU_SOFTIRQ , rcu_process_callbacks); + +/* +* We don't need protection against CPU-hotplug here because +* this is called early in boot, before either interrupts +* or the scheduler are operational. +*/ +cpu_notifier (rcu_cpu_notify,0 ); +for_each_online_cpu (i) +rcu_cpu_notify (NULL ,CPU_UP_PREPARE , (void *)(long )i); }#include "rcutree_plugin.h"
[tip:core/rcu] rcu: Re-arrange code to reduce #ifdef pain
In reply to Paul E. McKenneyCommit-ID: 6ebb237bece23275d1da149b61a342f0d4d06a08 Gitweb: http://git.kernel.org/tip/6ebb237bece23275d1da149b61a342f0d4d06a08 Author: Paul E. McKenney <[email protected]> AuthorDate: Sun, 22 Nov 2009 08:53:50 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sun, 22 Nov 2009 18:58:16 +0100
rcu: Re-arrange code to reduce #ifdef pain
Remove #ifdefs from kernel/rcupdate.c and include/linux/rcupdate.h by moving code to include/linux/rcutiny.h, include/linux/rcutree.h, and kernel/rcutree.c.
Also remove some definitions that are no longer used.
LKML-Reference: <1258908830885-git-send-email->
--- include/linux/rcupdate.h | 12 ----- include/linux/rcutiny.h | 11 +++++ include/linux/rcutree.h | 4 +- kernel/rcupdate.c | 104 ---------------------------------------------- kernel/rcutree.c | 80 +++++++++++++++++++++++++++++++++++ kernel/rcutree_plugin.h | 24 +++++++++++ 6 files changed, 118 insertions(+), 117 deletions(-)
include/linux/rcupdate.h +0 -12
@@ -52,11 +52,6 @@ struct rcu_head { };/* Exported common interfaces */ -#ifdef CONFIG_TREE_PREEMPT_RCU -extern void synchronize_rcu (void ); -#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */ -#define synchronize_rcu synchronize_sched -#endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */ extern void synchronize_rcu_bh (void );extern void synchronize_sched (void );extern void rcu_barrier (void );
@@ -67,13 +62,6 @@ extern int sched_expedited_torture_stats(char *page);/* Internal to kernel */ extern void rcu_init (void ); -extern void rcu_scheduler_starting (void ); -#ifndef CONFIG_TINY_RCU -extern int rcu_needs_cpu (int cpu); -#else -static inline int rcu_needs_cpu (int cpu) {return 0 ; } -#endif -extern int rcu_scheduler_active;#if defined(CONFIG_TREE_RCU ) || defined(CONFIG_TREE_PREEMPT_RCU )#include <linux/rcutree.h>
include/linux/rcutiny.h +11 -0
@@ -39,6 +39,11 @@ void rcu_bh_qs(int cpu);#define rcu_init_sched () do { } while (0)extern void rcu_check_callbacks (int cpu,int user); +static inline int rcu_needs_cpu (int cpu) +{ +return 0 ; +} +/* * Return the number of grace periods. */
@@ -57,6 +62,8 @@ static inline long rcu_batches_completed_bh(void)extern int rcu_expedited_torture_stats (char *page); +#define synchronize_rcu synchronize_sched +static inline void synchronize_rcu_expedited (void ) {synchronize_sched ();
@@ -86,6 +93,10 @@ static inline void rcu_exit_nohz(void)#endif /* #else #ifdef CONFIG_NO_HZ */ +static inline void rcu_scheduler_starting (void ) +{ +} +static inline void exit_rcu (void ) { }
include/linux/rcutree.h +3 -1
@@ -35,12 +35,14 @@ struct notifier_block;extern void rcu_sched_qs (int cpu);extern void rcu_bh_qs (int cpu);extern int rcu_needs_cpu (int cpu); +extern void rcu_scheduler_starting (void );extern int rcu_expedited_torture_stats (char *page);#ifdef CONFIG_TREE_PREEMPT_RCU extern void __rcu_read_lock (void );extern void __rcu_read_unlock (void ); +extern void synchronize_rcu (void );extern void exit_rcu (void );#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
@@ -55,7 +57,7 @@ static inline void __rcu_read_unlock(void)preempt_enable (); } -#define __synchronize_sched () synchronize_rcu() +#define synchronize_rcu synchronize_schedstatic inline void exit_rcu (void ) {
kernel/rcupdate.c +0 -104
119 lines not on this page · raw message
kernel/rcutree.c +80 -0
104 lines not on this page · raw message
kernel/rcutree_plugin.h +24 -0
@@ -425,6 +425,30 @@ void call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) }EXPORT_SYMBOL_GPL (call_rcu); +/** +* synchronize_rcu - wait until a grace period has elapsed. +* +* Control will return to the caller some time after a full grace +* period has elapsed, in other words after all currently executing RCU +* read-side critical sections have completed. RCU read-side critical +* sections are delimited by rcu_read_lock() and rcu_read_unlock(), +* and may be nested. +*/ +void synchronize_rcu (void ) +{ +struct rcu_synchronize rcu; + +if (!rcu_scheduler_active) +return ; + +init_completion (&rcu.completion); +/* Will wake me after RCU finished. */ +call_rcu (&rcu.head, wakeme_after_rcu); +/* Wait for it. */ +wait_for_completion (&rcu.completion); +} +EXPORT_SYMBOL_GPL (synchronize_rcu); +/* * Wait for an rcu-preempt grace period. We are supposed to expedite the * grace period, but this is the crude slow compatability hack, so just