Skip to contentSkip to conversation

RCU changes for v3.6

GIT PULL Ingo Molnar 1 participant 1 message Jul 22, 2012

Subsystems (8)
lore
Share & tools
lore

Ingo Molnar

author maintainer
#0 lore raw

Linus,

Please pull the latest RCU git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core-rcu-for-linus

HEAD: 5c09d127a112a78f95572921af88224f4091eb44 Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu

Quoting from Paul, the major features of this series are:

1. Preventing latency spikes of more than 200 microseconds for
kernels built with NR_CPUS=4096, which is reportedly becoming the default for some distros. This is a first step, as it does not help with systems that actually -have- 4096 CPUs (work on this case is in progress, but is not yet ready for mainline). This category also includes improving concurrency of rcu_barrier(), placed here due to conflicts. Posted to LKML at:
https://lkml.org/lkml/2012/6/22/381. Note that patches 18-22 of that series have been defered to 3.7, as they have not yet proven themselves to be mainline-ready (and yes, these are the ones intended to get rid of RCU's latency spikes for systems that actually have 4096 CPUs).

2. Updates to documentation and rcutorture fixes, the latter category
including improvements to rcu_barrier() testing. Posted to LKML at:
http://lkml.indiana.edu/hypermail/linux/kernel/1206.1/04094.html.

3. Miscellaneous fixes posted to LKML at:
https://lkml.org/lkml/2012/6/22/500, with the exception of the last commit, which was posted here:
http://www.gossamer-threads.com/lists/linux/kernel/1561830

4. RCU_FAST_NO_HZ fixes and improvements. Posted to LKML at:
http://lkml.indiana.edu/hypermail/linux/kernel/1206.1/00006.html
and http://www.gossamer-threads.com/lists/linux/kernel/1561833.
The first four patches of the first series went into 3.5 to fix a regression.

5. Code-style fixes. These were posted to LKML at
http://lkml.indiana.edu/hypermail/linux/kernel/1205.2/01180.html and
http://lkml.indiana.edu/hypermail/linux/kernel/1205.2/01181.html.

Thanks,

Ingo

------------------>
Carsten Emde (1):
rcu: Prevent uninitialized string in RCU CPU stall info

Paul E. McKenney (48):
Revert "rcu: Move PREEMPT_RCU preemption to switch_to() invocation"
rcu: Control RCU_FANOUT_LEAF from boot-time parameter
rcu: Four-level hierarchy is no longer experimental
rcu: Size rcu_node tree from nr_cpu_ids rather than NR_CPUS
rcu: Prevent excessive line length in RCU_STATE_INITIALIZER()
rcu: Place pointer to call_rcu() in rcu_data structure
rcu: Move _rcu_barrier()'s rcu_head structures to rcu_data structures
rcu: Move rcu_barrier_cpu_count to rcu_state structure
rcu: Move rcu_barrier_completion to rcu_state structure
rcu: Move rcu_barrier_mutex to rcu_state structure
rcu: Remove needless initialization
rcu: Increase rcu_barrier() concurrency
rcu: Add tracing for _rcu_barrier()
rcu: Add rcu_barrier() statistics to debugfs tracing
rcu: Remove unneeded __rcu_process_callbacks() argument
rcu: Introduce for_each_rcu_flavor() and use it
rcu: Use for_each_rcu_flavor() in TREE_RCU tracing
rcu: RCU_SAVE_DYNTICK code no longer ever dead
rcu: Update documentation to cover call_srcu() and srcu_barrier().
rcu: Rationalize ordering of torture_ops list
rcu: Test srcu_barrier() from rcutorture test suite
rcu: Fix bug in rcu_barrier() torture test
rcu: Fix diagnostic-printk typo in rcutorture
rcu: Make rcutorture fakewriters invoke rcu_barrier()
rcu: Fix detection of abruptly-ending stall
rcu: Consolidate duplicate callback-list initialization
rcu: Add ACCESS_ONCE() to ->qlen accesses
rcu: Add a gcc-style structure initializer for RCU pointers
rcu: Use new RCU_POINTER_INITIALIZER for gcc-style initializations
rcu: Remove return value from RCU_INIT_POINTER()
key: Remove extraneous parentheses from rcu_assign_keypointer()
rcu: Remove return value from rcu_assign_pointer()
rcu: Consolidate tree/tiny __rcu_read_{,un}lock() implementations
rcu: Remove function versions of __kfree_rcu and __is_kfree_rcu_offset
rcu: Make __call_rcu() handle invocation from idle
rcu: Prevent __call_rcu() from invoking RCU core on offline CPUs
rcu: Split RCU core processing out of __call_rcu()
rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU
rcu: Disable preemption in rcu_blocking_is_gp()
rcu: Add check for CPUs going offline with callbacks queued
rcu: Dump only the current CPU's buffers for idle-entry/exit warnings
rcu: The rcu_needs_cpu() function is not a quiescent state
rcu: Round FAST_NO_HZ lazy timeout to nearest second
rcu: Fix qlen_lazy breakage
rcu: Make RCU_FAST_NO_HZ respect nohz= boot parameter
rcu: Introduce check for callback list/count mismatch
rcu: Fix code-style issues involving "else"
rcu: Fix broken strings in RCU's source code.

 Documentation/RCU/checklist.txt     |   39 +--
 Documentation/RCU/rcubarrier.txt    |   15 +-
 Documentation/RCU/torture.txt       |    9 +
 Documentation/RCU/whatisRCU.txt     |    6 +-
 Documentation/kernel-parameters.txt |    5 +
 arch/um/drivers/mconsole_kern.c     |    1 -
 include/linux/init_task.h           |    4 +-
 include/linux/key.h                 |    4 +-
 include/linux/rcupdate.h            |   54 ++--
 include/linux/rcutiny.h             |    6 +
 include/linux/sched.h               |   10 -
 include/trace/events/rcu.h          |   45 ++++
 kernel/rcupdate.c                   |   44 ++++
 kernel/rcutiny.c                    |    4 +-
 kernel/rcutiny_plugin.h             |   56 +---
 kernel/rcutorture.c                 |   72 ++++--
 kernel/rcutree.c                    |  479 +++++++++++++++++++++++------------
 kernel/rcutree.h                    |   47 ++--
 kernel/rcutree_plugin.h             |  237 ++++-------------
 kernel/rcutree_trace.c              |  148 ++++++-----
 kernel/sched/core.c                 |    1 -
 kernel/time/tick-sched.c            |    2 +-
 lib/list_debug.c                    |    6 +-
 23 files changed, 688 insertions(+), 606 deletions(-)
Documentation/RCU/checklist.txt +20 -19
index 5c8d749..fc103d7 100644
@@ -162,9 +162,9 @@ over a rather long period of time, but improvements are always welcome!
 		when publicizing a pointer to a structure that can
 		be traversed by an RCU read-side critical section.
 
-5.	If call_rcu(), or a related primitive such as call_rcu_bh() or
-	call_rcu_sched(), is used, the callback function must be
-	written to be called from softirq context.  In particular,
+5.	If call_rcu(), or a related primitive such as call_rcu_bh(),
+	call_rcu_sched(), or call_srcu() is used, the callback function
+	must be written to be called from softirq context.  In particular,
 	it cannot block.
 
 6.	Since synchronize_rcu() can block, it cannot be called from
@@ -202,11 +202,12 @@ over a rather long period of time, but improvements are always welcome!
 	updater uses call_rcu_sched() or synchronize_sched(), then
 	the corresponding readers must disable preemption, possibly
 	by calling rcu_read_lock_sched() and rcu_read_unlock_sched().
-	If the updater uses synchronize_srcu(), the the corresponding
-	readers must use srcu_read_lock() and srcu_read_unlock(),
-	and with the same srcu_struct.	The rules for the expedited
-	primitives are the same as for their non-expedited counterparts.
-	Mixing things up will result in confusion and broken kernels.
+	If the updater uses synchronize_srcu() or call_srcu(),
+	the the corresponding readers must use srcu_read_lock() and
+	srcu_read_unlock(), and with the same srcu_struct.  The rules for
+	the expedited primitives are the same as for their non-expedited
+	counterparts.  Mixing things up will result in confusion and
+	broken kernels.
 
 	One exception to this rule: rcu_read_lock() and rcu_read_unlock()
 	may be substituted for rcu_read_lock_bh() and rcu_read_unlock_bh()
@@ -333,14 +334,14 @@ over a rather long period of time, but improvements are always welcome!
 	victim CPU from ever going offline.)
 
 14.	SRCU (srcu_read_lock(), srcu_read_unlock(), srcu_dereference(),
-	synchronize_srcu(), and synchronize_srcu_expedited()) may only
-	be invoked from process context.  Unlike other forms of RCU, it
-	-is- permissible to block in an SRCU read-side critical section
-	(demarked by srcu_read_lock() and srcu_read_unlock()), hence the
-	"SRCU": "sleepable RCU".  Please note that if you don't need
-	to sleep in read-side critical sections, you should be using
-	RCU rather than SRCU, because RCU is almost always faster and
-	easier to use than is SRCU.
+	synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu())
+	may only be invoked from process context.  Unlike other forms of
+	RCU, it -is- permissible to block in an SRCU read-side critical
+	section (demarked by srcu_read_lock() and srcu_read_unlock()),
+	hence the "SRCU": "sleepable RCU".  Please note that if you
+	don't need to sleep in read-side critical sections, you should be
+	using RCU rather than SRCU, because RCU is almost always faster
+	and easier to use than is SRCU.
 
 	If you need to enter your read-side critical section in a
 	hardirq or exception handler, and then exit that same read-side
@@ -353,8 +354,8 @@ over a rather long period of time, but improvements are always welcome!
 	cleanup_srcu_struct().	These are passed a "struct srcu_struct"
 	that defines the scope of a given SRCU domain.	Once initialized,
 	the srcu_struct is passed to srcu_read_lock(), srcu_read_unlock()
-	synchronize_srcu(), and synchronize_srcu_expedited().  A given
-	synchronize_srcu() waits only for SRCU read-side critical
+	synchronize_srcu(), synchronize_srcu_expedited(), and call_srcu().
+	A given synchronize_srcu() waits only for SRCU read-side critical
 	sections governed by srcu_read_lock() and srcu_read_unlock()
 	calls that have been passed the same srcu_struct.  This property
 	is what makes sleeping read-side critical sections tolerable --
@@ -374,7 +375,7 @@ over a rather long period of time, but improvements are always welcome!
 	requiring SRCU's read-side deadlock immunity or low read-side
 	realtime latency.
 
-	Note that, rcu_assign_pointer() relates to SRCU just as they do
+	Note that, rcu_assign_pointer() relates to SRCU just as it does
 	to other forms of RCU.
 
 15.	The whole point of call_rcu(), synchronize_rcu(), and friends
Documentation/RCU/rcubarrier.txt +4 -11
index e439a0e..38428c1 100644
@@ -79,8 +79,6 @@ complete. Pseudo-code using rcu_barrier() is as follows:
    2. Execute rcu_barrier().
    3. Allow the module to be unloaded.
 
-Quick Quiz #1: Why is there no srcu_barrier()?
-
 The rcutorture module makes use of rcu_barrier in its exit function
 as follows:
 
@@ -162,7 +160,7 @@ for any pre-existing callbacks to complete.
 Then lines 55-62 print status and do operation-specific cleanup, and
 then return, permitting the module-unload operation to be completed.
 
-Quick Quiz #2: Is there any other situation where rcu_barrier() might
+Quick Quiz #1: Is there any other situation where rcu_barrier() might
 	be required?
 
 Your module might have additional complications. For example, if your
@@ -242,7 +240,7 @@ reaches zero, as follows:
  4 complete(&rcu_barrier_completion);
  5 }
 
-Quick Quiz #3: What happens if CPU 0's rcu_barrier_func() executes
+Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes
 	immediately (thus incrementing rcu_barrier_cpu_count to the
 	value one), but the other CPU's rcu_barrier_func() invocations
 	are delayed for a full grace period? Couldn't this result in
@@ -259,12 +257,7 @@ so that your module may be safely unloaded.
 
 Answers to Quick Quizzes
 
-Quick Quiz #1: Why is there no srcu_barrier()?
-
-Answer: Since there is no call_srcu(), there can be no outstanding SRCU
-	callbacks. Therefore, there is no need to wait for them.
-
-Quick Quiz #2: Is there any other situation where rcu_barrier() might
+Quick Quiz #1: Is there any other situation where rcu_barrier() might
 	be required?
 
 Answer: Interestingly enough, rcu_barrier() was not originally
@@ -278,7 +271,7 @@ Answer: Interestingly enough, rcu_barrier() was not originally
 	implementing rcutorture, and found that rcu_barrier() solves
 	this problem as well.
 
-Quick Quiz #3: What happens if CPU 0's rcu_barrier_func() executes
+Quick Quiz #2: What happens if CPU 0's rcu_barrier_func() executes
 	immediately (thus incrementing rcu_barrier_cpu_count to the
 	value one), but the other CPU's rcu_barrier_func() invocations
 	are delayed for a full grace period? Couldn't this result in
Documentation/RCU/torture.txt +9 -0
index 4ddf391..7dce8a1 100644
@@ -174,11 +174,20 @@ torture_type	The type of RCU to test, with string values as follows:
 			and synchronize_rcu_bh_expedited().
 
 		"srcu": srcu_read_lock(), srcu_read_unlock() and
+			call_srcu().
+
+		"srcu_sync": srcu_read_lock(), srcu_read_unlock() and
 			synchronize_srcu().
 
 		"srcu_expedited": srcu_read_lock(), srcu_read_unlock() and
 			synchronize_srcu_expedited().
 
+		"srcu_raw": srcu_read_lock_raw(), srcu_read_unlock_raw(),
+			and call_srcu().
+
+		"srcu_raw_sync": srcu_read_lock_raw(), srcu_read_unlock_raw(),
+			and synchronize_srcu().
+
 		"sched": preempt_disable(), preempt_enable(), and
 			call_rcu_sched().
 
Documentation/RCU/whatisRCU.txt +3 -3
index 6bbe8dc..69ee188 100644
@@ -833,9 +833,9 @@ sched:	Critical sections	Grace period		Barrier
 
 SRCU:	Critical sections	Grace period		Barrier
 
-	srcu_read_lock		synchronize_srcu	N/A
-	srcu_read_unlock	synchronize_srcu_expedited
-	srcu_read_lock_raw
+	srcu_read_lock		synchronize_srcu	srcu_barrier
+	srcu_read_unlock	call_srcu
+	srcu_read_lock_raw	synchronize_srcu_expedited
 	srcu_read_unlock_raw
 	srcu_dereference
 
Documentation/kernel-parameters.txt +5 -0
index a92c5eb..12783fa 100644
@@ -2367,6 +2367,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			Set maximum number of finished RCU callbacks to process
 			in one batch.
 
+	rcutree.fanout_leaf=	[KNL,BOOT]
+			Increase the number of CPUs assigned to each
+			leaf rcu_node structure.  Useful for very large
+			systems.
+
 	rcutree.qhimark=	[KNL,BOOT]
 			Set threshold of queued
 			RCU callbacks over which batch limiting is disabled.
arch/um/drivers/mconsole_kern.c +0 -1
index 88e466b..43b39d6 100644
@@ -705,7 +705,6 @@ static void stack_proc(void *arg)
 	struct task_struct *from = current, *to = arg;
 
 	to->thread.saved_task = from;
-	rcu_switch_from(from);
 	switch_to(from, to, from);
 }
 
include/linux/init_task.h +2 -2
index 9e65eff..8a74761 100644
@@ -168,8 +168,8 @@ extern struct cred init_cred;
 	.children	= LIST_HEAD_INIT(tsk.children),			\
 	.sibling	= LIST_HEAD_INIT(tsk.sibling),			\
 	.group_leader	= &tsk,						\
-	RCU_INIT_POINTER(.real_cred, &init_cred),			\
-	RCU_INIT_POINTER(.cred, &init_cred),				\
+	RCU_POINTER_INITIALIZER(real_cred, &init_cred),			\
+	RCU_POINTER_INITIALIZER(cred, &init_cred),			\
 	.comm		= INIT_TASK_COMM,				\
 	.thread		= INIT_THREAD,					\
 	.fs		= &init_fs,					\
include/linux/key.h +3 -1
index 4cd22ed..cef3b31 100644
@@ -303,7 +303,9 @@ static inline bool key_is_instantiated(const struct key *key)
 				   rwsem_is_locked(&((struct key *)(KEY))->sem)))
 
 #define rcu_assign_keypointer(KEY, PAYLOAD)				\
-	(rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD))
+do {									\
+	rcu_assign_pointer((KEY)->payload.rcudata, (PAYLOAD));		\
+} while (0)
 
 #ifdef CONFIG_SYSCTL
 extern ctl_table key_sysctls[];
include/linux/rcupdate.h +20 -34

111 lines not on this page · raw message

include/linux/rcutiny.h +6 -0
index 854dc4c..4e56a9c 100644
@@ -87,6 +87,10 @@ static inline void kfree_call_rcu(struct rcu_head *head,
 
 #ifdef CONFIG_TINY_RCU
 
+static inline void rcu_preempt_note_context_switch(void)
+{
+}
+
 static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
 {
 	*delta_jiffies = ULONG_MAX;
@@ -95,6 +99,7 @@ static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
 
 #else /* #ifdef CONFIG_TINY_RCU */
 
+void rcu_preempt_note_context_switch(void);
 int rcu_preempt_needs_cpu(void);
 
 static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
@@ -108,6 +113,7 @@ static inline int rcu_needs_cpu(int cpu, unsigned long *delta_jiffies)
 static inline void rcu_note_context_switch(int cpu)
 {
 	rcu_sched_qs(cpu);
+	rcu_preempt_note_context_switch();
 }
 
 /*
include/linux/sched.h +0 -10
index 4059c0f..06a4c5f 100644
@@ -1871,22 +1871,12 @@ static inline void rcu_copy_process(struct task_struct *p)
 	INIT_LIST_HEAD(&p->rcu_node_entry);
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-	if (prev->rcu_read_lock_nesting != 0)
-		rcu_preempt_note_context_switch();
-}
-
 #else
 
 static inline void rcu_copy_process(struct task_struct *p)
 {
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-}
-
 #endif
 
 #ifdef CONFIG_SMP
include/trace/events/rcu.h +45 -0

57 lines not on this page · raw message

kernel/rcupdate.c +44 -0

50 lines not on this page · raw message

kernel/rcutiny.c +2 -2
index 37a5444..547b1fe 100644
@@ -172,7 +172,7 @@ void rcu_irq_enter(void)
 	local_irq_restore(flags);
 }
 
-#ifdef CONFIG_PROVE_RCU
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
 
 /*
  * Test whether RCU thinks that the current CPU is idle.
@@ -183,7 +183,7 @@ int rcu_is_cpu_idle(void)
 }
 EXPORT_SYMBOL(rcu_is_cpu_idle);
 
-#endif /* #ifdef CONFIG_PROVE_RCU */
+#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
 
 /*
  * Test whether the current CPU was interrupted from idle.  Nested
kernel/rcutiny_plugin.h +5 -51

98 lines not on this page · raw message

kernel/rcutorture.c +44 -28

183 lines not on this page · raw message

kernel/rcutree.c +310 -169

827 lines not on this page · raw message

kernel/rcutree.h +30 -17

153 lines not on this page · raw message

kernel/rcutree_plugin.h +55 -182

422 lines not on this page · raw message

kernel/rcutree_trace.c +78 -70

237 lines not on this page · raw message

kernel/sched/core.c +0 -1
index d5594a4..eaead2d 100644
@@ -2081,7 +2081,6 @@ context_switch(struct rq *rq, struct task_struct *prev,
 #endif
 
 	/* Here we just switch the register state and the stack. */
-	rcu_switch_from(prev);
 	switch_to(prev, next, prev);
 
 	barrier();
kernel/time/tick-sched.c +1 -1
index 8699978..66ff07f 100644
@@ -105,7 +105,7 @@ static ktime_t tick_init_jiffy_update(void)
 /*
  * NO HZ enabled ?
  */
-static int tick_nohz_enabled __read_mostly  = 1;
+int tick_nohz_enabled __read_mostly  = 1;
 
 /*
  * Enable / Disable tickless mode
lib/list_debug.c +2 -4
index 23a5e03..c24c2f7 100644
@@ -87,12 +87,10 @@ void __list_add_rcu(struct list_head *new,
 		    struct list_head *prev, struct list_head *next)
 {
 	WARN(next->prev != prev,
-		"list_add_rcu corruption. next->prev should be "
-		"prev (%p), but was %p. (next=%p).\n",
+		"list_add_rcu corruption. next->prev should be prev (%p), but was %p. (next=%p).\n",
 		prev, next->prev, next);
 	WARN(prev->next != next,
-		"list_add_rcu corruption. prev->next should be "
-		"next (%p), but was %p. (prev=%p).\n",
+		"list_add_rcu corruption. prev->next should be next (%p), but was %p. (prev=%p).\n",
 		next, prev->next, prev);
 	new->next = next;
 	new->prev = prev;