v2 Fixups for 3.6
Subsystems (2)
cover v2 Fixups for 3.6
Hello!
This patch series has general fixups and improvements. This is version 2, the prior version may be found at: https://lkml.org/lkml/2012/6/15/388.
The individual patches are as follows:
1. Get rid of the false positives from the code that recognizes
a CPU stall that ends just as it is detected.
2. Consolidate open-coded initializations of RCU callback lists.
3. Protect unsynchronized accesses to ->qlen with ACCESS_ONCE()
to document the unsynchronized access and to prevent compiler mischief.
4. Add RCU_POINTER_INITIALIZER() for static initialization to allow
do-while formulation of the existing RCU_INIT_POINTER().
5. Convert clever use of RCU_INIT_POINTER() for gcc-style
initialization to RCU_POINTER_INITIALIZER().
6. Wrap RCU_INIT_POINTER() macro in obligatory do-while.
7. Remove extraneous parentheses from rcu_assign_keypointer()
to allow rcu_assign_pointer() to be wrapped in do-while.
8. Wrap rcu_assign_pointer() macro in obligatory do-while.
9. Consolidate the identical CONFIG_TREE_PREEMPT_RCU and
CONFIG_TINY_PREEMPT_RCU versions of __rcu_read_lock() and __rcu_read_unlock(). 10. Remove the old inline-function version of __kfree_rcu() and __is_kfree_rcu_offset().
11. Make __call_rcu() correctly handle calls from the idle loop.
12. Make __call_rcu() correctly handle calls from a CPU that is
in the process of going offline.
13. Split __call_rcu() into the part that actually deals with
enqueueing the callback and the part that kicks RCU's grace-period machinery.
14. Make RCU_TINY's rcu_is_cpu_idle() function safe for "make
randconfig".
15. Properly initialize the preemptible-RCU string in kernels built
without CONFIG_RCU_FAST_NO_HZ (courtesy of Carsten Emde).
16. Disable preemption in rcu_blocking_is_gp() to allow future
optimization of preemptible-RCU synchronize_rcu() for single-CPU systems.
17. Add a warning for CPUs going offline with RCU callbacks queued.
Changes since version 1:
o Applied Josh Triplett review comments.
o Added patches 15-17.
Thanx, Paul
b/include/linux/init_task.h | 4 - b/include/linux/key.h | 2 b/include/linux/rcupdate.h | 8 ++ b/kernel/rcupdate.c | 44 +++++++++++ b/kernel/rcutiny.c | 4 - b/kernel/rcutiny_plugin.h | 47 ------------ b/kernel/rcutree.c | 4 - b/kernel/rcutree_plugin.h | 47 ------------ include/linux/rcupdate.h | 42 ++--------- kernel/rcutree.c | 164 ++++++++++++++++++++++---------------------- kernel/rcutree_plugin.h | 1 11 files changed, 157 insertions(+), 210 deletions(-)
PATCH 1/17 rcu: Fix detection of abruptly-ending stall
From: "Paul E. McKenney" <[email protected]>
The code that attempts to identify stalls that end just as we detect them is broken by both flavors of initialization failure. This commit therefore properly initializes and computes the count of the number of reasons why the RCU grace period is stalled.
--- kernel/rcutree.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/rcutree.c +2 -2
@@ -732,7 +732,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)int cpu;long delta;unsigned long flags; -int ndetected; +int ndetected =0 ;struct rcu_node *rnp =rcu_get_root (rsp);/* Only let one CPU complain about others per time interval. */
@@ -773,7 +773,7 @@ static void print_other_cpu_stall(struct rcu_state *rsp)*/ rnp =rcu_get_root (rsp);raw_spin_lock_irqsave (&rnp->lock, flags); - ndetected =rcu_print_task_stall (rnp); + ndetected +=rcu_print_task_stall (rnp);raw_spin_unlock_irqrestore (&rnp->lock, flags);print_cpu_stall_info_end ();
Signature · 1 line
1.7.8
PATCH 2/17 rcu: Consolidate duplicate callback-list initialization
From: "Paul E. McKenney" <[email protected]>
There are a couple of open-coded initializations of the rcu_data structure's RCU callback list. This commit therefore consolidates them into a new init_callback_list() function.
--- kernel/rcutree.c | 23 ++++++++++++++--------- 1 files changed, 14 insertions(+), 9 deletions(-)
kernel/rcutree.c +14 -9
@@ -936,6 +936,18 @@ check_for_new_grace_period(struct rcu_state *rsp, struct rcu_data *rdp) }/* +* Initialize the specified rcu_data structure's callback list to empty. +*/ +static void init_callback_list (struct rcu_data *rdp) +{ +int i; + + rdp->nxtlist =NULL ; +for (i =0 ; i <RCU_NEXT_SIZE ; i++) + rdp->nxttail[i] = &rdp->nxtlist; +} + +/* * Advance this CPU's callbacks, but only if the current grace period * has ended. This may be called only from the CPU to whom the rdp * belongs. In addition, the corresponding leaf rcu_node structure's
@@ -1327,8 +1339,6 @@ static void rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp, struct rcu_node *rnp, struct rcu_data *rdp) { - int i; -/* * Orphan the callbacks. First adjust the counts. This is safe * because ->onofflock excludes _rcu_barrier()'s adoption of
@@ -1368,9 +1378,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp, }/* Finally, initialize the rcu_data structure's list to empty. */ - rdp->nxtlist =NULL ; -for (i =0 ; i <RCU_NEXT_SIZE ; i++) - rdp->nxttail[i] = &rdp->nxtlist; +init_callback_list (rdp); }/*
@@ -2404,16 +2412,13 @@ static void __init rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) { unsigned long flags; -int i;struct rcu_data *rdp =per_cpu_ptr (rsp->rda, cpu);struct rcu_node *rnp =rcu_get_root (rsp);/* Set up local state, ensuring consistent view of global state. */ raw_spin_lock_irqsave (&rnp->lock, flags); rdp->grpmask =1UL << (cpu - rdp->mynode->grplo); - rdp->nxtlist =NULL ; -for (i =0 ; i <RCU_NEXT_SIZE ; i++) - rdp->nxttail[i] = &rdp->nxtlist; +init_callback_list (rdp); rdp->qlen_lazy =0 ; rdp->qlen =0 ; rdp->dynticks = &per_cpu (rcu_dynticks, cpu);
Signature · 1 line
1.7.8
PATCH 3/17 rcu: Add ACCESS_ONCE() to ->qlen accesses
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The _rcu_barrier() function accesses other CPUs' rcu_data structure's ->qlen field without benefit of locking. This commit therefore adds the required ACCESS_ONCE() wrappers around accesses and updates that need it.
ACCESS_ONCE() is not needed when a CPU accesses its own ->qlen, or in code that cannot run while _rcu_barrier() is sampling ->qlen fields.
--- kernel/rcutree.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/rcutree.c +4 -4
@@ -1349,7 +1349,7 @@ rcu_send_cbs_to_orphanage(int cpu, struct rcu_state *rsp, rsp->qlen += rdp->qlen; rdp->n_cbs_orphaned += rdp->qlen; rdp->qlen_lazy =0 ; - rdp->qlen =0 ; +ACCESS_ONCE (rdp->qlen) =0 ; }/*
@@ -1597,7 +1597,7 @@ static void rcu_do_batch(struct rcu_state *rsp, struct rcu_data *rdp) }smp_mb ();/* List handling before counting for rcu_barrier(). */ rdp->qlen_lazy -= count_lazy; - rdp->qlen -= count; +ACCESS_ONCE (rdp->qlen) -= count; rdp->n_cbs_invoked += count;/* Reinstate batch limit if we have worked down the excess. */
@@ -1886,7 +1886,7 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), rdp =this_cpu_ptr (rsp->rda);/* Add the callback to our list. */ - rdp->qlen++; +ACCESS_ONCE (rdp->qlen)++;if (lazy) rdp->qlen_lazy++;else
@@ -2420,7 +2420,7 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) rdp->grpmask =1UL << (cpu - rdp->mynode->grplo);init_callback_list (rdp); rdp->qlen_lazy =0 ; - rdp->qlen =0 ; +ACCESS_ONCE (rdp->qlen) =0 ; rdp->dynticks = &per_cpu (rcu_dynticks, cpu);WARN_ON_ONCE (rdp->dynticks->dynticks_nesting !=DYNTICK_TASK_EXIT_IDLE );WARN_ON_ONCE (atomic_read (&rdp->dynticks->dynticks) !=1 );
Signature · 1 line
1.7.8
PATCH 4/17 rcu: Add a gcc-style structure initializer for RCU pointers
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
RCU_INIT_POINTER() returns a value that is never used, and which should be abolished due to terminal ugliness:
q = RCU_INIT_POINTER(global_p, p);
However, there are two uses that cannot be handled by a do-while formulation because they do gcc-style initialization:
RCU_INIT_POINTER(.real_cred, &init_cred),
RCU_INIT_POINTER(.cred, &init_cred),
This usage is clever, but not necessarily the nicest approach. This commit therefore creates an RCU_POINTER_INITIALIZER() macro that is specifically designed for gcc-style initialization.
--- include/linux/rcupdate.h | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
include/linux/rcupdate.h +8 -0
@@ -906,6 +906,14 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)#define RCU_INIT_POINTER (p, v) \ p = (typeof(*v) __force __rcu *)(v) +/** +* RCU_POINTER_INITIALIZER() - statically initialize an RCU protected pointer +* +* GCC-style initialization for an RCU-protected pointer in a structure field. +*/ +#define RCU_POINTER_INITIALIZER (p, v) \ + .p = (typeof(*v) __force __rcu *)(v) + static __always_inline bool __is_kfree_rcu_offset(unsigned long offset) {return offset <4096 ;
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 04/17] rcu: Add a gcc-style structure initializer for RCU pointers
Quoted 8 lines · Paul E. McKenney
RCU_INIT_POINTER()returns a value that is never used, and which should be abolished due to terminal ugliness:q = RCU_INIT_POINTER(global_p, p);However, there are two uses that cannot be handled by a do-while formulation because they do gcc-style initialization:
Quoted 8 lines
RCU_INIT_POINTER(.real_cred, &init_cred),
RCU_INIT_POINTER(.cred, &init_cred),This usage is clever, but not necessarily the nicest approach. This commit therefore creates an
RCU_POINTER_INITIALIZER()macro that is specifically designed for gcc-style initialization.Signed-off-by Paul E. McKenney <[email protected]>
PATCH 5/17 rcu: Use new RCU_POINTER_INITIALIZER for gcc-style initializations
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit applies the INIT_RCU_POINTER() macro to all uses of RCU_POINTER_INITIALIZER() that were all too cleverly creating gcc-style initializations.
--- include/linux/init_task.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
include/linux/init_task.h +2 -2
@@ -168,8 +168,8 @@ extern struct cred init_cred; .children = LIST_HEAD_INIT(tsk.children), \ .sibling = LIST_HEAD_INIT(tsk.sibling), \ .group_leader = &tsk, \ - RCU_INIT_POINTER(.real_cred, &init_cred), \ - RCU_INIT_POINTER(.cred, &init_cred), \ +RCU_POINTER_INITIALIZER (real_cred, &init_cred), \ +RCU_POINTER_INITIALIZER (cred, &init_cred), \ .comm = INIT_TASK_COMM, \ .thread = INIT_THREAD, \ .fs = &init_fs, \
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 05/17] rcu: Use new RCU_POINTER_INITIALIZER for gcc-style initializations
Quoted 7 lines · Paul E. McKenney
From: "Paul E. McKenney" <[email protected]>
This commit applies the
INIT_RCU_POINTER()macro to all uses ofRCU_POINTER_INITIALIZER()that were all too cleverly creating gcc-style initializations.Signed-off-by Paul E. McKenney <[email protected]>
PATCH 6/17 rcu: Remove return value from RCU_INIT_POINTER()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The return value from RCU_INIT_POINTER() is not used, and using it would be quite ugly, for example:
q = RCU_INIT_POINTER(global_p, p);
To prevent this sort of ugliness from appearing, this commit wraps RCU_INIT_POINTER() in a do-while loop.
--- include/linux/rcupdate.h | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +3 -1
@@ -904,7 +904,9 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)* the reader-accessible portions of the linked structure. */ #define RCU_INIT_POINTER (p, v) \ - p = (typeof(*v) __force __rcu *)(v) + do { \ + p = (typeof(*v) __force __rcu *)(v); \ + } while (0)/** * RCU_POINTER_INITIALIZER() - statically initialize an RCU protected pointer
Signature · 1 line
1.7.8
Re: [PATCH tip/core/rcu 06/17] rcu: Remove return value from RCU_INIT_POINTER()
Quoted 5 lines · Paul E. McKenney
From: "Paul E. McKenney" <[email protected]>
The return value from
RCU_INIT_POINTER()is not used, and using it would be quite ugly, for example:
Quoted 7 lines
q = RCU_INIT_POINTER(global_p, p);To prevent this sort of ugliness from appearing, this commit wraps
RCU_INIT_POINTER()in a do-while loop.Signed-off-by Paul E. McKenney <[email protected]>Reviewed-by Josh Triplett <[email protected]>
PATCH 7/17 key: Remove extraneous parentheses from rcu_assign_keypointer()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit removes the extraneous parentheses from rcu_assign_keypointer() so that rcu_assign_pointer() can be wrapped in do-while.
--- include/linux/key.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/key.h +1 -1
@@ -303,7 +303,7 @@ static inline bool key_is_instantiated(const struct key *key) rwsem_is_locked(&((struct key *)(KEY))->sem)))#define rcu_assign_keypointer (KEY ,PAYLOAD ) \ - (rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD)) + rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD)#ifdef CONFIG_SYSCTL extern ctl_table key_sysctls[];
Signature · 1 line
1.7.8
David Howells
maintainerRe: [PATCH tip/core/rcu 07/17] key: Remove extraneous parentheses from rcu_assign_keypointer()
Quoted 2 lines · Paul E. McKenney
This commit removes the extraneous parentheses from
rcu_assign_keypointer()so thatrcu_assign_pointer()can be wrapped in do-while.
Should it be wrapped in a do-while itself then?
And whilst you're at it, could you wrap PAYLOAD in parentheses?
Thanks,
David
Re: [PATCH tip/core/rcu 07/17] key: Remove extraneous parentheses from rcu_assign_keypointer()
Quoted 8 lines · David Howells
Quoted 2 lines · Paul E. McKenney
This commit removes the extraneous parentheses from
rcu_assign_keypointer()so thatrcu_assign_pointer()can be wrapped in do-while.Should it be wrapped in a do-while itself then?
And whilst you're at it, could you wrap PAYLOAD in parentheses?
Will do!
Thanx, Paul
Paul E. McKenney
Reviewed-by Josh TriplettRe: [PATCH tip/core/rcu 07/17] key: Remove extraneous parentheses from rcu_assign_keypointer()
Quoted 11 lines · Paul E. McKenney
Quoted 8 lines · David Howells
Quoted 2 lines · Paul E. McKenney
This commit removes the extraneous parentheses from
rcu_assign_keypointer()so thatrcu_assign_pointer()can be wrapped in do-while.Should it be wrapped in a do-while itself then?
And whilst you're at it, could you wrap PAYLOAD in parentheses?
Will do!
Before I forget again... Is this what you had in mind?
Thanx, Paul
------------------------------------------------------------------------
key: Remove extraneous parentheses from rcu_assign_keypointer()
This commit removes the extraneous parentheses from rcu_assign_keypointer() so that rcu_assign_pointer() can be wrapped in do-while. It also wraps rcu_assign_keypointer() in a do-while and parenthesizes its final argument, as suggested by David Howells.
include/linux/key.h +3 -1
@@ -303,7 +303,9 @@ static inline bool key_is_instantiated(const struct key *key) rwsem_is_locked(&((struct key *)(KEY))->sem)))#define rcu_assign_keypointer (KEY ,PAYLOAD ) \ - (rcu_assign_pointer((KEY)->payload.rcudata, PAYLOAD)) +do { \ + rcu_assign_pointer((KEY)->payload.rcudata, (PAYLOAD)) \ +while (0)#ifdef CONFIG_SYSCTL extern ctl_table key_sysctls[];
PATCH 8/17 rcu: Remove return value from rcu_assign_pointer()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The return value from rcu_assign_pointer() is not used, and using it would be quite ugly, for example:
q = rcu_assign_pointer(global_p, p);
To prevent this sort of ugliness from spreading, this commit wraps rcu_assign_pointer() in a do-while loop.
--- include/linux/rcupdate.h | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
include/linux/rcupdate.h +3 -3
@@ -514,10 +514,10 @@ static inline void rcu_preempt_sleep_check(void) (_________p1); \ })#define __rcu_assign_pointer (p, v, space) \ - ({ \ + do { \ smp_wmb(); \ (p) = (typeof(*v) __force space *)(v); \ - }) + } while (0)/**
@@ -852,7 +852,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)* * Assigns the specified value to the specified RCU-protected * pointer, ensuring that any concurrent RCU readers will see -* any prior initialization. Returns the value assigned. +* any prior initialization. * * Inserts memory barriers on architectures that require them * (which is most of them), and also prevents the compiler from
Signature · 1 line
1.7.8
PATCH 9/17 rcu: Consolidate tree/tiny __rcu_read_{,un}lock() implementations
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The CONFIG_TREE_PREEMPT_RCU and CONFIG_TINY_PREEMPT_RCU versions of __rcu_read_lock() and __rcu_read_unlock() are identical, so this commit consolidates them into kernel/rcupdate.h.
--- include/linux/rcupdate.h | 1 + kernel/rcupdate.c | 44 +++++++++++++++++++++++++++++++++++++++++++ kernel/rcutiny_plugin.h | 47 +--------------------------------------------- kernel/rcutree_plugin.h | 47 +--------------------------------------------- 4 files changed, 47 insertions(+), 92 deletions(-)
include/linux/rcupdate.h +1 -0
@@ -147,6 +147,7 @@ extern void synchronize_sched(void);extern void __rcu_read_lock (void );extern void __rcu_read_unlock (void ); +extern void rcu_read_unlock_special (struct task_struct *t);void synchronize_rcu (void );/*
kernel/rcupdate.c +44 -0
50 lines not on this page · raw message
kernel/rcutiny_plugin.h +1 -46
69 lines not on this page · raw message
kernel/rcutree_plugin.h +1 -46
71 lines not on this page · raw message
Signature · 1 line
1.7.8
PATCH 10/17 rcu: Remove function versions of __kfree_rcu and __is_kfree_rcu_offset
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Commit d8169d4c (Make __kfree_rcu() less dependent on compiler choices) added cpp macro versions of __kfree_rcu() and __is_kfree_rcu_offset(), but failed to remove the old inline-function versions. This commit does this cleanup.
--- include/linux/rcupdate.h | 18 ------------------ 1 files changed, 0 insertions(+), 18 deletions(-)
include/linux/rcupdate.h +0 -18
@@ -917,24 +917,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)#define RCU_POINTER_INITIALIZER (p, v) \ .p = (typeof(*v) __force __rcu *)(v) -static __always_inline bool __is_kfree_rcu_offset(unsigned long offset) -{ -return offset <4096 ; -} - -static __always_inline -void __kfree_rcu(struct rcu_head *head, unsigned long offset) -{ -typedef void (*rcu_callback )(struct rcu_head *); - -BUILD_BUG_ON (!__builtin_constant_p (offset)); - -/* See the kfree_rcu() header comment. */ -BUILD_BUG_ON (!__is_kfree_rcu_offset (offset)); - -kfree_call_rcu (head, (rcu_callback )offset); -} -/* * Does the specified offset indicate that the corresponding rcu_head * structure can be handled by kfree_rcu()?
Signature · 1 line
1.7.8
PATCH 11/17 rcu: Make __call_rcu() handle invocation from idle
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Although __call_rcu() is handled correctly when called from a momentary non-idle period, if it is called on a CPU that RCU believes to be idle on RCU_FAST_NO_HZ kernels, the callback might be indefinitely postponed. This commit therefore ensures that RCU is aware of the new callback and has a chance to force the CPU out of dyntick-idle mode when a new callback is posted.
--- include/linux/rcupdate.h | 13 ++++--------- kernel/rcutree.c | 15 +++++++++------ 2 files changed, 13 insertions(+), 15 deletions(-)
include/linux/rcupdate.h +4 -9
@@ -257,6 +257,10 @@ static inline void destroy_rcu_head_on_stack(struct rcu_head *head) }#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */ +#if defined(CONFIG_DEBUG_LOCK_ALLOC ) || defined(CONFIG_SMP ) +extern int rcu_is_cpu_idle (void ); +#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SMP) */ +#if defined(CONFIG_HOTPLUG_CPU ) && defined(CONFIG_PROVE_RCU )bool rcu_lockdep_current_cpu_online (void );#else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
@@ -268,15 +272,6 @@ static inline bool rcu_lockdep_current_cpu_online(void)#ifdef CONFIG_DEBUG_LOCK_ALLOC -#ifdef CONFIG_PROVE_RCU -extern int rcu_is_cpu_idle (void ); -#else /* !CONFIG_PROVE_RCU */ -static inline int rcu_is_cpu_idle (void ) -{ -return 0 ; -} -#endif /* else !CONFIG_PROVE_RCU */ -static inline void rcu_lock_acquire (struct lockdep_map *map) {lock_acquire (map,0 ,0 ,2 ,1 ,NULL , _THIS_IP_);
kernel/rcutree.c +9 -6
@@ -584,8 +584,6 @@ void rcu_nmi_exit(void)WARN_ON_ONCE (atomic_read (&rdtp->dynticks) &0x1 ); } -#ifdef CONFIG_PROVE_RCU -/** * rcu_is_cpu_idle - see if RCU thinks that the current CPU is idle *
@@ -603,7 +601,7 @@ int rcu_is_cpu_idle(void) }EXPORT_SYMBOL (rcu_is_cpu_idle); -#ifdef CONFIG_HOTPLUG_CPU +#if defined(CONFIG_PROVE_RCU ) && defined(CONFIG_HOTPLUG_CPU )/* * Is the current CPU online? Disable preemption to avoid false positives
@@ -644,9 +642,7 @@ bool rcu_lockdep_current_cpu_online(void) }EXPORT_SYMBOL_GPL (rcu_lockdep_current_cpu_online); -#endif /* #ifdef CONFIG_HOTPLUG_CPU */ - -#endif /* #ifdef CONFIG_PROVE_RCU */ +#endif /* #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) */ /** * rcu_is_cpu_rrupt_from_idle - see if idle or immediately interrupted from idle
@@ -1901,6 +1897,13 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),else trace_rcu_callback (rsp->name, head, rdp->qlen_lazy, rdp->qlen); +/* +* If called from an extended quiescent state, invoke the RCU +* core in order to force a re-evaluation of RCU's idleness. +*/ +if (rcu_is_cpu_idle ()) +invoke_rcu_core (); +/* If interrupts were disabled, don't dive into RCU core. */ if (irqs_disabled_flags (flags)) {local_irq_restore (flags);
Signature · 1 line
1.7.8
PATCH 12/17 rcu: Prevent __call_rcu() from invoking RCU core on offline CPUs
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The __call_rcu() function will invoke the RCU core, for example, if it detects that the current CPU has too many callbacks. However, this can happen on an offline CPU that is on its way to the idle loop, in which case it is an error to invoke the RCU core, and the excess callbacks will be adopted in any case. This commit therefore adds checks to __call_rcu() for running on an offline CPU, refraining from invoking the RCU core in this case.
--- kernel/rcutree.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
kernel/rcutree.c +3 -3
@@ -1901,11 +1901,11 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),* If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness. */ -if (rcu_is_cpu_idle ()) +if (rcu_is_cpu_idle () &&cpu_online (smp_processor_id ()))invoke_rcu_core (); -/* If interrupts were disabled, don't dive into RCU core. */ -if (irqs_disabled_flags (flags)) { +/* If interrupts were disabled or CPU offline, don't invoke RCU core. */ +if (irqs_disabled_flags (flags) ||cpu_is_offline (smp_processor_id ())) {local_irq_restore (flags);return ; }
Signature · 1 line
1.7.8
PATCH 13/17 rcu: Split RCU core processing out of __call_rcu()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The __call_rcu() function is a bit overweight, so this commit splits it into actual enqueuing of and accounting for the callback (__call_rcu()) and associated RCU-core processing (__call_rcu_core()).
--- kernel/rcutree.c | 90 +++++++++++++++++++++++++++++------------------------ 1 files changed, 49 insertions(+), 41 deletions(-)
kernel/rcutree.c +49 -41
@@ -1858,45 +1858,12 @@ static void invoke_rcu_core(void)raise_softirq (RCU_SOFTIRQ ); } -static void -__call_rcu (struct rcu_head *head,void (*func)(struct rcu_head *rcu), -struct rcu_state *rsp,bool lazy) +/* +* Handle any core-RCU processing required by a call_rcu() invocation. +*/ +static void __call_rcu_core (struct rcu_state *rsp,struct rcu_data *rdp, +struct rcu_head *head,unsigned long flags) { -unsigned long flags; -struct rcu_data *rdp; - -WARN_ON_ONCE ((unsigned long )head &0x3 );/* Misaligned rcu_head! */ -debug_rcu_head_queue (head); - head->func = func; - head->next =NULL ; - -smp_mb ();/* Ensure RCU update seen before callback registry. */ - -/* -* Opportunistically note grace-period endings and beginnings. -* Note that we might see a beginning right after we see an -* end, but never vice versa, since this CPU has to pass through -* a quiescent state betweentimes. -*/ -local_irq_save (flags); - rdp =this_cpu_ptr (rsp->rda); - -/* Add the callback to our list. */ -ACCESS_ONCE (rdp->qlen)++; -if (lazy) - rdp->qlen_lazy++; -else -rcu_idle_count_callbacks_posted (); -smp_mb ();/* Count before adding callback for rcu_barrier(). */ - *rdp->nxttail[RCU_NEXT_TAIL ] = head; - rdp->nxttail[RCU_NEXT_TAIL ] = &head->next; - -if (__is_kfree_rcu_offset ((unsigned long )func)) -trace_rcu_kfree_callback (rsp->name, head, (unsigned long )func, - rdp->qlen_lazy, rdp->qlen); -else -trace_rcu_callback (rsp->name, head, rdp->qlen_lazy, rdp->qlen); -/* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness.
@@ -1905,10 +1872,8 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu),invoke_rcu_core ();/* If interrupts were disabled or CPU offline, don't invoke RCU core. */ -if (irqs_disabled_flags (flags) ||cpu_is_offline (smp_processor_id ())) { -local_irq_restore (flags); +if (irqs_disabled_flags (flags) ||cpu_is_offline (smp_processor_id ()))return ; - }/* * Force the grace period if too many callbacks or too long waiting.
@@ -1941,6 +1906,49 @@ __call_rcu(struct rcu_head *head, void (*func)(struct rcu_head *rcu), } }else if (ULONG_CMP_LT (ACCESS_ONCE (rsp->jiffies_force_qs), jiffies))force_quiescent_state (rsp,1 ); +} + +static void +__call_rcu (struct rcu_head *head,void (*func)(struct rcu_head *rcu), +struct rcu_state *rsp,bool lazy) +{ +unsigned long flags; +struct rcu_data *rdp; + +WARN_ON_ONCE ((unsigned long )head &0x3 );/* Misaligned rcu_head! */ +debug_rcu_head_queue (head); + head->func = func; + head->next =NULL ; + +smp_mb ();/* Ensure RCU update seen before callback registry. */ + +/* +* Opportunistically note grace-period endings and beginnings. +* Note that we might see a beginning right after we see an +* end, but never vice versa, since this CPU has to pass through +* a quiescent state betweentimes. +*/ +local_irq_save (flags); + rdp =this_cpu_ptr (rsp->rda); + +/* Add the callback to our list. */ +ACCESS_ONCE (rdp->qlen)++; +if (lazy) + rdp->qlen_lazy++; +else +rcu_idle_count_callbacks_posted (); +smp_mb ();/* Count before adding callback for rcu_barrier(). */ + *rdp->nxttail[RCU_NEXT_TAIL ] = head; + rdp->nxttail[RCU_NEXT_TAIL ] = &head->next; + +if (__is_kfree_rcu_offset ((unsigned long )func)) +trace_rcu_kfree_callback (rsp->name, head, (unsigned long )func, + rdp->qlen_lazy, rdp->qlen); +else +trace_rcu_callback (rsp->name, head, rdp->qlen_lazy, rdp->qlen); + +/* Go handle any RCU core processing required. */ +__call_rcu_core (rsp, rdp, head, flags);local_irq_restore (flags); }
Signature · 1 line
1.7.8
PATCH 14/17 rcu: Fix rcu_is_cpu_idle() #ifdef in TINY_RCU
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The rcu_is_cpu_idle() function is used if CONFIG_DEBUG_LOCK_ALLOC, but TINY_RCU defines it only when CONFIG_PROVE_RCU. This causes build failures when CONFIG_DEBUG_LOCK_ALLOC=y but CONFIG_PROVE_RCU=n. This commit therefore adjusts the #ifdefs for rcu_is_cpu_idle() so that it is defined when CONFIG_DEBUG_LOCK_ALLOC=y.
--- kernel/rcutiny.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
kernel/rcutiny.c +2 -2
@@ -172,7 +172,7 @@ void rcu_irq_enter(void)local_irq_restore (flags); } -#ifdef CONFIG_PROVE_RCU +#ifdef CONFIG_DEBUG_LOCK_ALLOC /* * Test whether RCU thinks that the current CPU is idle.
@@ -183,7 +183,7 @@ int rcu_is_cpu_idle(void) }EXPORT_SYMBOL (rcu_is_cpu_idle); -#endif /* #ifdef CONFIG_PROVE_RCU */ +#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ /* * Test whether the current CPU was interrupted from idle. Nested
Signature · 1 line
1.7.8
PATCH 15/17 Prevent uninitialized string in RCU stall info
In reply to Paul E. McKenneyFrom: Carsten Emde <[email protected]>
An uninitialized string may be displayed at the end of the rcu_preempt detected stall info such as
0: (1 GPs behind) idle=075/140000000000000/0 =8?^D=8?^D
^^^^^^^^^^
if CONFIG_RCU_FAST_NO_HZ is not defined.
This trivial patch clears the string in this case.
--- kernel/rcutree_plugin.h | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
kernel/rcutree_plugin.h +1 -0
@@ -2205,6 +2205,7 @@ static void print_cpu_stall_fast_no_hz(char *cp, int cpu)static void print_cpu_stall_fast_no_hz (char *cp,int cpu) { + *cp ='\0' ; }#endif /* #else #ifdef CONFIG_RCU_FAST_NO_HZ */
Signature · 1 line
1.7.8
PATCH 16/17 rcu: Disable preemption in rcu_blocking_is_gp()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
It is time to optimize CONFIG_TREE_PREEMPT_RCU's synchronize_rcu() for uniprocessor optimization, which means that rcu_blocking_is_gp() can no longer rely on RCU read-side critical sections having disabled preemption. This commit therefore disables preemption across rcu_blocking_is_gp()'s scan of the cpu_online_mask.
--- kernel/rcutree.c | 19 ++----------------- 1 files changed, 2 insertions(+), 17 deletions(-)
kernel/rcutree.c +2 -17
@@ -1978,28 +1978,13 @@ EXPORT_SYMBOL_GPL(call_rcu_bh);* occasionally incorrectly indicate that there are multiple CPUs online * when there was in fact only one the whole time, as this just adds * some overhead: RCU still operates correctly. -* -* Of course, sampling num_online_cpus() with preemption enabled can -* give erroneous results if there are concurrent CPU-hotplug operations. -* For example, given a demonic sequence of preemptions in num_online_cpus() -* and CPU-hotplug operations, there could be two or more CPUs online at -* all times, but num_online_cpus() might well return one (or even zero). -* -* However, all such demonic sequences require at least one CPU-offline -* operation. Furthermore, rcu_blocking_is_gp() giving the wrong answer -* is only a problem if there is an RCU read-side critical section executing -* throughout. But RCU-sched and RCU-bh read-side critical sections -* disable either preemption or bh, which prevents a CPU from going offline. -* Therefore, the only way that rcu_blocking_is_gp() can incorrectly return -* that there is only one CPU when in fact there was more than one throughout -* is when there were no RCU readers in the system. If there are no -* RCU readers, the grace period by definition can be of zero length, -* regardless of the number of online CPUs. */ static inline int rcu_blocking_is_gp (void ) {might_sleep ();/* Check for RCU read-side critical section. */ +preempt_disable ();return num_online_cpus () <=1 ; +preempt_enable (); }/**
Signature · 1 line
1.7.8
PATCH 17/17 rcu: Add check for CPUs going offline with callbacks queued
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
If a CPU goes offline with callbacks queued, those callbacks might be indefinitely postponed, which can result in a system hang. This commit therefore inserts warnings for this condition.
--- kernel/rcutree.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/rcutree.c +3 -0
@@ -1506,6 +1506,9 @@ static void rcu_cleanup_dead_cpu(int cpu, struct rcu_state *rsp)raw_spin_unlock_irqrestore (&rnp->lock, flags);if (need_report &RCU_OFL_TASKS_EXP_GP )rcu_report_exp_rnp (rsp, rnp, true); +WARN_ONCE (rdp->qlen !=0 || rdp->nxtlist !=NULL , +"rcu_cleanup_dead_cpu: Callbacks on offline CPU %d: qlen=%lu, nxtlist=%p\n" , + cpu, rdp->qlen, rdp->nxtlist); }#else /* #ifdef CONFIG_HOTPLUG_CPU */
Signature · 1 line
1.7.8