rcu: fixes for lockdep RCU and accelerated dyntick GPs
cover rcu: fixes for lockdep RCU and accelerated dyntick GPs
Hello!
This patchset includes three fixes for problems in tip/core/rcu:
1. Convert grace-period acceleration for the last non-dynticked
RCU to a trivial state machine in order to avoid illegally invoking __rcu_process_callbacks() with irqs disabled.
2. Make the non-PROVE_RCU variant of rcu_read_lock_sched_held()
understand that preemption is disabled during boot before the scheduler starts. (The earlier patch did the PROVE_RCU variant, but missed the non-PROVE_RCU variant.)
3. Add some irq-disabling to the grace-period acceleration
in #1 above and also enforce a hold-off period so that the last non-dynticked CPU doesn't softirq itself to death when there are multiple RCU callbacks in flight.
Thanx, Paul
b/include/linux/rcupdate.h | 3 - b/kernel/rcutree.c | 3 + b/kernel/rcutree.h | 1 b/kernel/rcutree_plugin.h | 74 ++++++++++++++++++++++++++++++++------------- kernel/rcutree_plugin.h | 11 +++++- 5 files changed, 67 insertions(+), 25 deletions(-)
PATCH 1/3 rcu: fixes for accelerated grace periods for last non-dynticked CPU
It is illegal to invoke __rcu_process_callbacks() with irqs disabled, so do it indirectly via raise_softirq(). This requires a state-machine implementation to cycle through the grace-period machinery the required number of times.
--- kernel/rcutree.c | 3 ++ kernel/rcutree.h | 1 + kernel/rcutree_plugin.h | 73 ++++++++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 20 deletions(-)
kernel/rcutree.c +3 -0
@@ -1341,6 +1341,9 @@ static void rcu_process_callbacks(struct softirq_action *unused)* grace-period manipulations above. */ smp_mb ();/* See above block comment. */ + +/* If we are last CPU on way to dyntick-idle mode, accelerate it. */ +rcu_needs_cpu_flush (); } static void
kernel/rcutree.h +1 -0
@@ -373,5 +373,6 @@ static int rcu_preempt_needs_cpu(int cpu);static void __cpuinit rcu_preempt_init_percpu_data (int cpu);static void rcu_preempt_send_cbs_to_orphanage (void );static void __init __rcu_init_preempt (void ); +static void rcu_needs_cpu_flush (void );#endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h +53 -20
101 lines not on this page · raw message
Signature · 1 line
1.6.6
[tip:core/rcu] rcu: Fix accelerated grace periods for last non-dynticked CPU
Commit-ID: a47cd880b50e14b0b6f5e9d426ae9a2676c9c474 Gitweb: http://git.kernel.org/tip/a47cd880b50e14b0b6f5e9d426ae9a2676c9c474 Author: Paul E. McKenney <[email protected]> AuthorDate: Fri, 26 Feb 2010 16:38:56 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 27 Feb 2010 09:53:52 +0100
rcu: Fix accelerated grace periods for last non-dynticked CPU
It is invalid to invoke __rcu_process_callbacks() with irqs disabled, so do it indirectly via raise_softirq(). This requires a state-machine implementation to cycle through the grace-period machinery the required number of times.
LKML-Reference: <[email protected]>
--- kernel/rcutree.c | 3 ++ kernel/rcutree.h | 1 + kernel/rcutree_plugin.h | 73 ++++++++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 20 deletions(-)
kernel/rcutree.c +3 -0
@@ -1341,6 +1341,9 @@ static void rcu_process_callbacks(struct softirq_action *unused)* grace-period manipulations above. */ smp_mb ();/* See above block comment. */ + +/* If we are last CPU on way to dyntick-idle mode, accelerate it. */ +rcu_needs_cpu_flush (); } static void
kernel/rcutree.h +1 -0
@@ -373,5 +373,6 @@ static int rcu_preempt_needs_cpu(int cpu);static void __cpuinit rcu_preempt_init_percpu_data (int cpu);static void rcu_preempt_send_cbs_to_orphanage (void );static void __init __rcu_init_preempt (void ); +static void rcu_needs_cpu_flush (void );#endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h +53 -20
101 lines not on this page · raw message
PATCH 2/3 rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot
In reply to Paul E. McKenneyBefore the scheduler starts, all tasks are non-preemptible by definition. So, during that time, rcu_read_lock_sched_held() needs to always return "true". This patch makes that be so for RCU_PROVE_LOCKING=n.
--- include/linux/rcupdate.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +1 -1
@@ -166,7 +166,7 @@ static inline int rcu_read_lock_bh_held(void)static inline int rcu_read_lock_sched_held (void ) { -return preempt_count () !=0 ; +return preempt_count () !=0 || !rcu_scheduler_active; }#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
Signature · 1 line
1.6.6
[tip:core/rcu] rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot
Commit-ID: 0b1c87278a8c7e394022ec184a0b44a3886b6fde Gitweb: http://git.kernel.org/tip/0b1c87278a8c7e394022ec184a0b44a3886b6fde Author: Paul E. McKenney <[email protected]> AuthorDate: Fri, 26 Feb 2010 16:38:57 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 27 Feb 2010 09:53:52 +0100
rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot
Before the scheduler starts, all tasks are non-preemptible by definition. So, during that time, rcu_read_lock_sched_held() needs to always return "true". This patch makes that be so for RCU_PROVE_LOCKING=n.
LKML-Reference: <[email protected]>
--- include/linux/rcupdate.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +1 -1
@@ -166,7 +166,7 @@ static inline int rcu_read_lock_bh_held(void)static inline int rcu_read_lock_sched_held (void ) { -return preempt_count () !=0 ; +return preempt_count () !=0 || !rcu_scheduler_active; }#endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
PATCH 3/3 rcu: more fixes for accelerated GPs for last non-dynticked CPU
In reply to Paul E. McKenneyThis patch disables irqs across the call to rcu_needs_cpu(). It also enforces a hold-off period so that the idle loop doesn't softirq itself to death when there are lots of RCU callbacks in flight on the last non-dynticked CPU.
--- kernel/rcutree_plugin.h | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-)
kernel/rcutree_plugin.h +9 -1
@@ -986,6 +986,7 @@ static void rcu_needs_cpu_flush(void)#define RCU_NEEDS_CPU_FLUSHES 5 static DEFINE_PER_CPU(int, rcu_dyntick_drain); +static DEFINE_PER_CPU(unsigned long, rcu_dyntick_holdoff);/* * Check to see if any future RCU-related work will need to be done
@@ -1013,6 +1014,7 @@ int rcu_needs_cpu(int cpu)for_each_cpu_not (thatcpu, nohz_cpu_mask)if (thatcpu != cpu) {per_cpu (rcu_dyntick_drain, cpu) =0 ; +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies -1 ;return rcu_needs_cpu_quick_check (cpu); }
@@ -1022,6 +1024,7 @@ int rcu_needs_cpu(int cpu)per_cpu (rcu_dyntick_drain, cpu) =RCU_NEEDS_CPU_FLUSHES ; }else if (--per_cpu (rcu_dyntick_drain, cpu) <=0 ) {/* We have hit the limit, so time to give up. */ +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies;return rcu_needs_cpu_quick_check (cpu); }
@@ -1038,8 +1041,10 @@ int rcu_needs_cpu(int cpu) }/* If RCU callbacks are still pending, RCU still needs this CPU. */ -if (c) +if (c) {raise_softirq (RCU_SOFTIRQ ); +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies; + }return c; }
@@ -1050,10 +1055,13 @@ int rcu_needs_cpu(int cpu)static void rcu_needs_cpu_flush (void ) {int cpu =smp_processor_id (); +unsigned long flags;if (per_cpu (rcu_dyntick_drain, cpu) <=0 )return ; +local_irq_save (flags); (void )rcu_needs_cpu (cpu); +local_irq_restore (flags); }#endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */
Signature · 1 line
1.6.6
[tip:core/rcu] rcu: Fix accelerated GPs for last non-dynticked CPU
Commit-ID: 71da81324c83ef65bb196c7f874ac1c6996d8287 Gitweb: http://git.kernel.org/tip/71da81324c83ef65bb196c7f874ac1c6996d8287 Author: Paul E. McKenney <[email protected]> AuthorDate: Fri, 26 Feb 2010 16:38:58 -0800 Committer: Ingo Molnar <[email protected]> CommitDate: Sat, 27 Feb 2010 09:53:53 +0100
rcu: Fix accelerated GPs for last non-dynticked CPU
This patch disables irqs across the call to rcu_needs_cpu(). It also enforces a hold-off period so that the idle loop doesn't softirq itself to death when there are lots of RCU callbacks in flight on the last non-dynticked CPU.
LKML-Reference: <[email protected]>
--- kernel/rcutree_plugin.h | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-)
kernel/rcutree_plugin.h +9 -1
@@ -986,6 +986,7 @@ static void rcu_needs_cpu_flush(void)#define RCU_NEEDS_CPU_FLUSHES 5 static DEFINE_PER_CPU(int, rcu_dyntick_drain); +static DEFINE_PER_CPU(unsigned long, rcu_dyntick_holdoff);/* * Check to see if any future RCU-related work will need to be done
@@ -1013,6 +1014,7 @@ int rcu_needs_cpu(int cpu)for_each_cpu_not (thatcpu, nohz_cpu_mask)if (thatcpu != cpu) {per_cpu (rcu_dyntick_drain, cpu) =0 ; +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies -1 ;return rcu_needs_cpu_quick_check (cpu); }
@@ -1022,6 +1024,7 @@ int rcu_needs_cpu(int cpu)per_cpu (rcu_dyntick_drain, cpu) =RCU_NEEDS_CPU_FLUSHES ; }else if (--per_cpu (rcu_dyntick_drain, cpu) <=0 ) {/* We have hit the limit, so time to give up. */ +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies;return rcu_needs_cpu_quick_check (cpu); }
@@ -1038,8 +1041,10 @@ int rcu_needs_cpu(int cpu) }/* If RCU callbacks are still pending, RCU still needs this CPU. */ -if (c) +if (c) {raise_softirq (RCU_SOFTIRQ ); +per_cpu (rcu_dyntick_holdoff, cpu) = jiffies; + }return c; }
@@ -1050,10 +1055,13 @@ int rcu_needs_cpu(int cpu)static void rcu_needs_cpu_flush (void ) {int cpu =smp_processor_id (); +unsigned long flags;if (per_cpu (rcu_dyntick_drain, cpu) <=0 )return ; +local_irq_save (flags); (void )rcu_needs_cpu (cpu); +local_irq_restore (flags); }#endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */