Skip to contentSkip to conversation

v4 somewhat-expedited "big hammer" RCU grace periods

RFC PATCH In review Paul E. McKenney 2 participants 3 messages May 8, 2009

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 v4 somewhat-expedited "big hammer" RCU grace periods

Fourth cut of "big hammer" expedited RCU grace periods. This uses a kthread that schedules itself on all online CPUs in turn, thus forcing a grace period. The synchronize_sched(), synchronize_rcu(), and synchronize_bh() primitives wake this kthread up and then wait for it to force the grace period.

As before, this does nothing to expedite callbacks already registered with call_rcu() or call_rcu_bh(), but there is no need to. Just maps to synchronize_rcu() and a new synchronize_rcu_bh() on preemptable RCU, which has more complex grace-period detection -- this can be fixed later.

Passes light rcutorture testing. Grace periods take around 200 microseconds on an 8-CPU Power machine. This is a good order of magnitude better than v3, but an order of magnitude slower than v2. Furthermore, it will get slower the more CPUs you have, and eight CPUs is not all that many these days. So this implementation still does not cut it.

Once again, I am posting this on the off-chance that I made some stupid mistake that someone might spot. Absent that, I am taking yet another different approach, namely setting up per-CPU threads that are awakened via smp_call_function(), permitting the quiescent states to be waited for in parallel.

Shortcomings:

o Too slow!!! Thinking in terms of using per-CPU kthreads.

o The wait_event() calls result in 120-second warnings, need
to use something like wait_event_interruptible(). There are probably other corner cases that need attention.

o Does not address preemptable RCU.

Changes since v3:

o Use a kthread that schedules itself on each CPU in turn to
force a grace period. The synchronize_rcu() primitive wakes up the kthread in order to avoid messing with affinity masks on user tasks.

o Tried a number of additional variations on the v3 approach, none
of which helped much.

Changes since v2:

o Use reschedule IPIs rather than a softirq.

Changes since v1:

o Added rcutorture support, and added exports required by
rcutorture.

o Added comment stating that smp_call_function() implies a
memory barrier, suggested by Mathieu.

o Added #include for delay.h.

---

 include/linux/rcuclassic.h |   16 +++
 include/linux/rcupdate.h   |   24 ++---
 include/linux/rcupreempt.h |   10 ++
 include/linux/rcutree.h    |   13 ++
 kernel/rcupdate.c          |  103 +++++++++++++++++++++++
 kernel/rcupreempt.c        |    1 
 kernel/rcutorture.c        |  200 ++++++++++++++++++++++++---------------------
 7 files changed, 261 insertions(+), 106 deletions(-)
include/linux/rcuclassic.h +15 -1
index bfd92e1..ea1ceb2 100644
@@ -158,14 +158,28 @@ extern struct lockdep_map rcu_lock_map;
 
 #define call_rcu_sched(head, func) call_rcu(head, func)
 
+static inline void synchronize_rcu_expedited(void)
+{
+	synchronize_sched_expedited();
+}
+
+static inline void synchronize_rcu_bh_expedited(void)
+{
+	synchronize_sched_expedited();
+}
+
 extern void __rcu_init(void);
-#define rcu_init_sched()	do { } while (0)
 extern void rcu_check_callbacks(int cpu, int user);
 extern void rcu_restart_cpu(int cpu);
 
 extern long rcu_batches_completed(void);
 extern long rcu_batches_completed_bh(void);
 
+static inline void rcu_init_sched(void)
+{
+	synchronize_sched_expedited_init();
+}
+
 #define rcu_enter_nohz()	do { } while (0)
 #define rcu_exit_nohz()		do { } while (0)
 
include/linux/rcupdate.h +12 -12
index 15fbb3c..46ddd78 100644
@@ -51,7 +51,18 @@ struct rcu_head {
 	void (*func)(struct rcu_head *head);
 };
 
-/* Internal to kernel, but needed by rcupreempt.h. */
+/* Exported common interfaces */
+extern void synchronize_rcu(void);
+extern void rcu_barrier(void);
+extern void rcu_barrier_bh(void);
+extern void rcu_barrier_sched(void);
+extern void synchronize_sched_expedited(void);
+
+/* Internal to kernel */
+extern void rcu_init(void);
+extern void rcu_scheduler_starting(void);
+extern void synchronize_sched_expedited_init(void);
+extern int rcu_needs_cpu(int cpu);
 extern int rcu_scheduler_active;
 
 #if defined(CONFIG_CLASSIC_RCU)
@@ -259,15 +270,4 @@ extern void call_rcu(struct rcu_head *head,
 extern void call_rcu_bh(struct rcu_head *head,
 			void (*func)(struct rcu_head *head));
 
-/* Exported common interfaces */
-extern void synchronize_rcu(void);
-extern void rcu_barrier(void);
-extern void rcu_barrier_bh(void);
-extern void rcu_barrier_sched(void);
-
-/* Internal to kernel */
-extern void rcu_init(void);
-extern void rcu_scheduler_starting(void);
-extern int rcu_needs_cpu(int cpu);
-
 #endif /* __LINUX_RCUPDATE_H */
include/linux/rcupreempt.h +10 -0
index fce5227..78117ed 100644
@@ -74,6 +74,16 @@ extern int rcu_needs_cpu(int cpu);
 
 extern void __synchronize_sched(void);
 
+static inline void synchronize_rcu_expedited(void)
+{
+	synchronize_rcu();  /* Placeholder for new rcupreempt implementation. */
+}
+
+static inline void synchronize_rcu_bh_expedited(void)
+{
+	synchronize_rcu();  /* Placeholder for new rcupreempt implementation. */
+}
+
 extern void __rcu_init(void);
 extern void rcu_init_sched(void);
 extern void rcu_check_callbacks(int cpu, int user);
include/linux/rcutree.h +12 -1
index 58b2aa5..7b533ec 100644
@@ -279,8 +279,14 @@ static inline void __rcu_read_unlock_bh(void)
 
 #define call_rcu_sched(head, func) call_rcu(head, func)
 
-static inline void rcu_init_sched(void)
+static inline void synchronize_rcu_expedited(void)
+{
+	synchronize_sched_expedited();
+}
+
+static inline void synchronize_rcu_bh_expedited(void)
 {
+	synchronize_sched_expedited();
 }
 
 extern void __rcu_init(void);
@@ -290,6 +296,11 @@ extern void rcu_restart_cpu(int cpu);
 extern long rcu_batches_completed(void);
 extern long rcu_batches_completed_bh(void);
 
+static inline void rcu_init_sched(void)
+{
+	synchronize_sched_expedited_init();
+}
+
 #ifdef CONFIG_NO_HZ
 void rcu_enter_nohz(void);
 void rcu_exit_nohz(void);
kernel/rcupdate.c +103 -0

121 lines not on this page · raw message

kernel/rcupreempt.c +1 -0
index ce97a4d..4485758 100644
@@ -1507,6 +1507,7 @@ void __init rcu_init_sched(void)
 						  NULL,
 						  "rcu_sched_grace_period");
 	WARN_ON(IS_ERR(rcu_sched_grace_period_task));
+	synchronize_sched_expedited_init();
 }
 
 #ifdef CONFIG_RCU_TRACE
kernel/rcutorture.c +108 -92

292 lines not on this page · raw message

#1 lore raw
Quoted 17 lines · Paul E. McKenney

Fourth cut of "big hammer" expedited RCU grace periods. This uses a kthread that schedules itself on all online CPUs in turn, thus forcing a grace period. The synchronize_sched(), synchronize_rcu(), and synchronize_bh() primitives wake this kthread up and then wait for it to force the grace period.

As before, this does nothing to expedite callbacks already registered with call_rcu() or call_rcu_bh(), but there is no need to. Just maps to synchronize_rcu() and a new synchronize_rcu_bh() on preemptable RCU, which has more complex grace-period detection -- this can be fixed later.

Passes light rcutorture testing. Grace periods take around 200 microseconds on an 8-CPU Power machine. This is a good order of magnitude better than v3, but an order of magnitude slower than v2. Furthermore, it will get slower the more CPUs you have, and eight CPUs is not all that many these days. So this implementation still does not cut it.

Quoted 6 lines

Once again, I am posting this on the off-chance that I made some stupid mistake that someone might spot. Absent that, I am taking yet another different approach, namely setting up per-CPU threads that are awakened via smp_call_function(), permitting the quiescent states to be waited for in parallel.

I dont know, dont we have possibility one cpu is dedicated for the use of a cpu hungry real time thread ?

krcu_sched_expedited() would dead lock or something ?

Quoted 53 lines

Shortcomings:

o Too slow!!! Thinking in terms of using per-CPU kthreads.

o The wait_event() calls result in 120-second warnings, need
to use something like wait_event_interruptible(). There are probably other corner cases that need attention.

o Does not address preemptable RCU.

Changes since v3:

o Use a kthread that schedules itself on each CPU in turn to
force a grace period. The synchronize_rcu() primitive wakes up the kthread in order to avoid messing with affinity masks on user tasks.

o Tried a number of additional variations on the v3 approach, none
of which helped much.

Changes since v2:

o Use reschedule IPIs rather than a softirq.

Changes since v1:

o Added rcutorture support, and added exports required by
rcutorture.

o Added comment stating that smp_call_function() implies a
memory barrier, suggested by Mathieu.

o Added #include for delay.h.

---

 include/linux/rcuclassic.h |   16 +++
 include/linux/rcupdate.h   |   24 ++---
 include/linux/rcupreempt.h |   10 ++
 include/linux/rcutree.h    |   13 ++
 kernel/rcupdate.c          |  103 +++++++++++++++++++++++
 kernel/rcupreempt.c        |    1 
 kernel/rcutorture.c        |  200 ++++++++++++++++++++++++---------------------
 7 files changed, 261 insertions(+), 106 deletions(-)
kernel/rcupdate.c +6 -0
+/*
+ * Kernel thread that processes synchronize_sched_expedited() requests.
+ * This is implemented as a separate kernel thread to avoid the need
+ * to mess with other tasks' cpumasks.
+ */
+static int krcu_sched_expedited(void *arg)

kernel/rcupdate.c static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self, · line 282

Quoted 10 lines
kernel/rcupdate.c +10 -0
+{
+	int cpu;
+
+	do {
+		wait_event(need_sched_expedited_wq, need_sched_expedited);
+		need_sched_expedited = 0;
+		get_online_cpus();
+		for_each_online_cpu(cpu) {
+			sched_setaffinity(0, &cpumask_of_cpu(cpu));
+			schedule();

<<no return>>

Quoted 7 lines
kernel/rcupdate.c +7 -0
+		}
+		put_online_cpus();
+		sched_expedited_done = 1;
+		wake_up(&sched_expedited_done_wq);
+	} while (!kthread_should_stop());
+	return 0;
+}
#2 lore raw
Quoted 25 lines · Eric Dumazet
Quoted 17 lines · Paul E. McKenney

Fourth cut of "big hammer" expedited RCU grace periods. This uses a kthread that schedules itself on all online CPUs in turn, thus forcing a grace period. The synchronize_sched(), synchronize_rcu(), and synchronize_bh() primitives wake this kthread up and then wait for it to force the grace period.

As before, this does nothing to expedite callbacks already registered with call_rcu() or call_rcu_bh(), but there is no need to. Just maps to synchronize_rcu() and a new synchronize_rcu_bh() on preemptable RCU, which has more complex grace-period detection -- this can be fixed later.

Passes light rcutorture testing. Grace periods take around 200 microseconds on an 8-CPU Power machine. This is a good order of magnitude better than v3, but an order of magnitude slower than v2. Furthermore, it will get slower the more CPUs you have, and eight CPUs is not all that many these days. So this implementation still does not cut it.

Quoted 6 lines

Once again, I am posting this on the off-chance that I made some stupid mistake that someone might spot. Absent that, I am taking yet another different approach, namely setting up per-CPU threads that are awakened via smp_call_function(), permitting the quiescent states to be waited for in parallel.

Quoted 4 lines

I dont know, dont we have possibility one cpu is dedicated for the use of a cpu hungry real time thread ?

krcu_sched_expedited() would dead lock or something ?

Good point!!!

One approach would be to use a prio-99 RT per-CPU thread that sleeps unless/until an expedited grace period is required.

Aggressive real-time workloads would need to avoid doing things (like changing networking configuration) that require expedited grace periods.

Seem reasonable?

Thanx, Paul

Quoted 75 lines
Quoted 53 lines

Shortcomings:

o Too slow!!! Thinking in terms of using per-CPU kthreads.

o The wait_event() calls result in 120-second warnings, need
to use something like wait_event_interruptible(). There are probably other corner cases that need attention.

o Does not address preemptable RCU.

Changes since v3:

o Use a kthread that schedules itself on each CPU in turn to
force a grace period. The synchronize_rcu() primitive wakes up the kthread in order to avoid messing with affinity masks on user tasks.

o Tried a number of additional variations on the v3 approach, none
of which helped much.

Changes since v2:

o Use reschedule IPIs rather than a softirq.

Changes since v1:

o Added rcutorture support, and added exports required by
rcutorture.

o Added comment stating that smp_call_function() implies a
memory barrier, suggested by Mathieu.

o Added #include for delay.h.

---

 include/linux/rcuclassic.h |   16 +++
 include/linux/rcupdate.h   |   24 ++---
 include/linux/rcupreempt.h |   10 ++
 include/linux/rcutree.h    |   13 ++
 kernel/rcupdate.c          |  103 +++++++++++++++++++++++
 kernel/rcupreempt.c        |    1 
 kernel/rcutorture.c        |  200 ++++++++++++++++++++++++---------------------
 7 files changed, 261 insertions(+), 106 deletions(-)
kernel/rcupdate.c +6 -0
+/*
+ * Kernel thread that processes synchronize_sched_expedited() requests.
+ * This is implemented as a separate kernel thread to avoid the need
+ * to mess with other tasks' cpumasks.
+ */
+static int krcu_sched_expedited(void *arg)

kernel/rcupdate.c static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self, · line 282

Quoted 10 lines
kernel/rcupdate.c +10 -0
+{
+	int cpu;
+
+	do {
+		wait_event(need_sched_expedited_wq, need_sched_expedited);
+		need_sched_expedited = 0;
+		get_online_cpus();
+		for_each_online_cpu(cpu) {
+			sched_setaffinity(0, &cpumask_of_cpu(cpu));
+			schedule();

<<no return>>

Quoted 7 lines
kernel/rcupdate.c +7 -0
+		}
+		put_online_cpus();
+		sched_expedited_done = 1;
+		wake_up(&sched_expedited_done_wq);
+	} while (!kthread_should_stop());
+	return 0;
+}