Skip to contentSkip to conversation

rcu: fixes for lockdep RCU and accelerated dyntick GPs

PATCH In review Paul E. McKenney 1 participant 7 messages Feb 27, 2010 RCU

lore
Share & tools
lore
#0 lore raw

cover rcu: fixes for lockdep RCU and accelerated dyntick GPs

Hello!

This patchset includes three fixes for problems in tip/core/rcu:

1. Convert grace-period acceleration for the last non-dynticked
RCU to a trivial state machine in order to avoid illegally invoking __rcu_process_callbacks() with irqs disabled.

2. Make the non-PROVE_RCU variant of rcu_read_lock_sched_held()
understand that preemption is disabled during boot before the scheduler starts. (The earlier patch did the PROVE_RCU variant, but missed the non-PROVE_RCU variant.)

3. Add some irq-disabling to the grace-period acceleration
in #1 above and also enforce a hold-off period so that the last non-dynticked CPU doesn't softirq itself to death when there are multiple RCU callbacks in flight.

Thanx, Paul

 b/include/linux/rcupdate.h |    3 -
 b/kernel/rcutree.c         |    3 +
 b/kernel/rcutree.h         |    1 
 b/kernel/rcutree_plugin.h  |   74 ++++++++++++++++++++++++++++++++-------------
 kernel/rcutree_plugin.h    |   11 +++++-
 5 files changed, 67 insertions(+), 25 deletions(-)
#1 lore raw

PATCH 1/3 rcu: fixes for accelerated grace periods for last non-dynticked CPU

It is illegal to invoke __rcu_process_callbacks() with irqs disabled, so do it indirectly via raise_softirq(). This requires a state-machine implementation to cycle through the grace-period machinery the required number of times.

---
 kernel/rcutree.c        |    3 ++
 kernel/rcutree.h        |    1 +
 kernel/rcutree_plugin.h |   73 ++++++++++++++++++++++++++++++++++-------------
 3 files changed, 57 insertions(+), 20 deletions(-)
kernel/rcutree.c +3 -0
index 335bfe4..3ec8160 100644
@@ -1341,6 +1341,9 @@ static void rcu_process_callbacks(struct softirq_action *unused)
 	 * grace-period manipulations above.
 	 */
 	smp_mb(); /* See above block comment. */
+
+	/* If we are last CPU on way to dyntick-idle mode, accelerate it. */
+	rcu_needs_cpu_flush();
 }
 
 static void
kernel/rcutree.h +1 -0
index 2ceb083..1439eb5 100644
@@ -373,5 +373,6 @@ static int rcu_preempt_needs_cpu(int cpu);
 static void __cpuinit rcu_preempt_init_percpu_data(int cpu);
 static void rcu_preempt_send_cbs_to_orphanage(void);
 static void __init __rcu_init_preempt(void);
+static void rcu_needs_cpu_flush(void);
 
 #endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h +53 -20

101 lines not on this page · raw message

Signature · 1 line
1.6.6
#2 lore raw

[tip:core/rcu] rcu: Fix accelerated grace periods for last non-dynticked CPU

Commit-ID:  a47cd880b50e14b0b6f5e9d426ae9a2676c9c474
Gitweb:     http://git.kernel.org/tip/a47cd880b50e14b0b6f5e9d426ae9a2676c9c474
Author:     Paul E. McKenney <[email protected]>
AuthorDate: Fri, 26 Feb 2010 16:38:56 -0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 27 Feb 2010 09:53:52 +0100

rcu: Fix accelerated grace periods for last non-dynticked CPU

It is invalid to invoke __rcu_process_callbacks() with irqs disabled, so do it indirectly via raise_softirq(). This requires a state-machine implementation to cycle through the grace-period machinery the required number of times.

LKML-Reference: <[email protected]>

---
 kernel/rcutree.c        |    3 ++
 kernel/rcutree.h        |    1 +
 kernel/rcutree_plugin.h |   73 ++++++++++++++++++++++++++++++++++-------------
 3 files changed, 57 insertions(+), 20 deletions(-)
kernel/rcutree.c +3 -0
index 335bfe4..3ec8160 100644
@@ -1341,6 +1341,9 @@ static void rcu_process_callbacks(struct softirq_action *unused)
 	 * grace-period manipulations above.
 	 */
 	smp_mb(); /* See above block comment. */
+
+	/* If we are last CPU on way to dyntick-idle mode, accelerate it. */
+	rcu_needs_cpu_flush();
 }
 
 static void
kernel/rcutree.h +1 -0
index 2ceb083..1439eb5 100644
@@ -373,5 +373,6 @@ static int rcu_preempt_needs_cpu(int cpu);
 static void __cpuinit rcu_preempt_init_percpu_data(int cpu);
 static void rcu_preempt_send_cbs_to_orphanage(void);
 static void __init __rcu_init_preempt(void);
+static void rcu_needs_cpu_flush(void);
 
 #endif /* #ifndef RCU_TREE_NONCORE */
kernel/rcutree_plugin.h +53 -20

101 lines not on this page · raw message

#3 lore raw

PATCH 2/3 rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot

In reply to Paul E. McKenney

Before the scheduler starts, all tasks are non-preemptible by definition. So, during that time, rcu_read_lock_sched_held() needs to always return "true". This patch makes that be so for RCU_PROVE_LOCKING=n.

---
 include/linux/rcupdate.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +1 -1
index fcea332..c843736 100644
@@ -166,7 +166,7 @@ static inline int rcu_read_lock_bh_held(void)
 
 static inline int rcu_read_lock_sched_held(void)
 {
-	return preempt_count() != 0;
+	return preempt_count() != 0 || !rcu_scheduler_active;
 }
 
 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
Signature · 1 line
1.6.6
#4 lore raw

[tip:core/rcu] rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot

Commit-ID:  0b1c87278a8c7e394022ec184a0b44a3886b6fde
Gitweb:     http://git.kernel.org/tip/0b1c87278a8c7e394022ec184a0b44a3886b6fde
Author:     Paul E. McKenney <[email protected]>
AuthorDate: Fri, 26 Feb 2010 16:38:57 -0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 27 Feb 2010 09:53:52 +0100

rcu: Make non-RCU_PROVE_LOCKING rcu_read_lock_sched_held() understand boot

Before the scheduler starts, all tasks are non-preemptible by definition. So, during that time, rcu_read_lock_sched_held() needs to always return "true". This patch makes that be so for RCU_PROVE_LOCKING=n.

LKML-Reference: <[email protected]>

---
 include/linux/rcupdate.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/linux/rcupdate.h +1 -1
index fcea332..c843736 100644
@@ -166,7 +166,7 @@ static inline int rcu_read_lock_bh_held(void)
 
 static inline int rcu_read_lock_sched_held(void)
 {
-	return preempt_count() != 0;
+	return preempt_count() != 0 || !rcu_scheduler_active;
 }
 
 #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */
#5 lore raw

PATCH 3/3 rcu: more fixes for accelerated GPs for last non-dynticked CPU

In reply to Paul E. McKenney

This patch disables irqs across the call to rcu_needs_cpu(). It also enforces a hold-off period so that the idle loop doesn't softirq itself to death when there are lots of RCU callbacks in flight on the last non-dynticked CPU.

---
 kernel/rcutree_plugin.h |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)
kernel/rcutree_plugin.h +9 -1
index ed241fc..464ad2c 100644
@@ -986,6 +986,7 @@ static void rcu_needs_cpu_flush(void)
 
 #define RCU_NEEDS_CPU_FLUSHES 5
 static DEFINE_PER_CPU(int, rcu_dyntick_drain);
+static DEFINE_PER_CPU(unsigned long, rcu_dyntick_holdoff);
 
 /*
  * Check to see if any future RCU-related work will need to be done
@@ -1013,6 +1014,7 @@ int rcu_needs_cpu(int cpu)
 	for_each_cpu_not(thatcpu, nohz_cpu_mask)
 		if (thatcpu != cpu) {
 			per_cpu(rcu_dyntick_drain, cpu) = 0;
+			per_cpu(rcu_dyntick_holdoff, cpu) = jiffies - 1;
 			return rcu_needs_cpu_quick_check(cpu);
 		}
 
@@ -1022,6 +1024,7 @@ int rcu_needs_cpu(int cpu)
 		per_cpu(rcu_dyntick_drain, cpu) = RCU_NEEDS_CPU_FLUSHES;
 	} else if (--per_cpu(rcu_dyntick_drain, cpu) <= 0) {
 		/* We have hit the limit, so time to give up. */
+		per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
 		return rcu_needs_cpu_quick_check(cpu);
 	}
 
@@ -1038,8 +1041,10 @@ int rcu_needs_cpu(int cpu)
 	}
 
 	/* If RCU callbacks are still pending, RCU still needs this CPU. */
-	if (c)
+	if (c) {
 		raise_softirq(RCU_SOFTIRQ);
+		per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
+	}
 	return c;
 }
 
@@ -1050,10 +1055,13 @@ int rcu_needs_cpu(int cpu)
 static void rcu_needs_cpu_flush(void)
 {
 	int cpu = smp_processor_id();
+	unsigned long flags;
 
 	if (per_cpu(rcu_dyntick_drain, cpu) <= 0)
 		return;
+	local_irq_save(flags);
 	(void)rcu_needs_cpu(cpu);
+	local_irq_restore(flags);
 }
 
 #endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */
Signature · 1 line
1.6.6
#6 lore raw

[tip:core/rcu] rcu: Fix accelerated GPs for last non-dynticked CPU

Commit-ID:  71da81324c83ef65bb196c7f874ac1c6996d8287
Gitweb:     http://git.kernel.org/tip/71da81324c83ef65bb196c7f874ac1c6996d8287
Author:     Paul E. McKenney <[email protected]>
AuthorDate: Fri, 26 Feb 2010 16:38:58 -0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 27 Feb 2010 09:53:53 +0100

rcu: Fix accelerated GPs for last non-dynticked CPU

This patch disables irqs across the call to rcu_needs_cpu(). It also enforces a hold-off period so that the idle loop doesn't softirq itself to death when there are lots of RCU callbacks in flight on the last non-dynticked CPU.

LKML-Reference: <[email protected]>

---
 kernel/rcutree_plugin.h |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)
kernel/rcutree_plugin.h +9 -1
index ed241fc..464ad2c 100644
@@ -986,6 +986,7 @@ static void rcu_needs_cpu_flush(void)
 
 #define RCU_NEEDS_CPU_FLUSHES 5
 static DEFINE_PER_CPU(int, rcu_dyntick_drain);
+static DEFINE_PER_CPU(unsigned long, rcu_dyntick_holdoff);
 
 /*
  * Check to see if any future RCU-related work will need to be done
@@ -1013,6 +1014,7 @@ int rcu_needs_cpu(int cpu)
 	for_each_cpu_not(thatcpu, nohz_cpu_mask)
 		if (thatcpu != cpu) {
 			per_cpu(rcu_dyntick_drain, cpu) = 0;
+			per_cpu(rcu_dyntick_holdoff, cpu) = jiffies - 1;
 			return rcu_needs_cpu_quick_check(cpu);
 		}
 
@@ -1022,6 +1024,7 @@ int rcu_needs_cpu(int cpu)
 		per_cpu(rcu_dyntick_drain, cpu) = RCU_NEEDS_CPU_FLUSHES;
 	} else if (--per_cpu(rcu_dyntick_drain, cpu) <= 0) {
 		/* We have hit the limit, so time to give up. */
+		per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
 		return rcu_needs_cpu_quick_check(cpu);
 	}
 
@@ -1038,8 +1041,10 @@ int rcu_needs_cpu(int cpu)
 	}
 
 	/* If RCU callbacks are still pending, RCU still needs this CPU. */
-	if (c)
+	if (c) {
 		raise_softirq(RCU_SOFTIRQ);
+		per_cpu(rcu_dyntick_holdoff, cpu) = jiffies;
+	}
 	return c;
 }
 
@@ -1050,10 +1055,13 @@ int rcu_needs_cpu(int cpu)
 static void rcu_needs_cpu_flush(void)
 {
 	int cpu = smp_processor_id();
+	unsigned long flags;
 
 	if (per_cpu(rcu_dyntick_drain, cpu) <= 0)
 		return;
+	local_irq_save(flags);
 	(void)rcu_needs_cpu(cpu);
+	local_irq_restore(flags);
 }
 
 #endif /* #else #if !defined(CONFIG_RCU_FAST_NO_HZ) */