Fix rcu_migrate_callback() to allow for multiple waiters

The current interaction between rcu_barrier() and CPU hotplug can result in multiple tasks waiting on rcu_migrate_wq: there can be one CPU-hotplug notifier, and, because rcu_barrier() invokes wait_migrated_callbacks() outside of rcu_barrier_mutex, an arbitrarily large number of tasks executing in _rcu_barrier(). This situation could result in hangs, because rcu_migrate_callback() would wake up but one task.

This patch addresses this problem by awakening all sleepers.

Located-by Mathieu Desnoyers <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

---

 rcupdate.c |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

kernel/rcupdate.c +5 -0

index bd5d5c8..6c59e1b 100644
@@ -211,10 +211,17 @@ void rcu_barrier_sched(void)
 }
 EXPORT_SYMBOL_GPL(rcu_barrier_sched);
 
+/*
+ * Wake up anyone waiting on migration of RCU callbacks from a CPU
+ * going offline.  There can be only one CPU-hotplug notifier waiting,
+ * but there can be any number of rcu_barrier() invocations waiting,
+ * due to the fact that rcu_barrier() does its wait outside of the

Quoted 30 lines · Paul E. McKenney

The current interaction between rcu_barrier() and CPU hotplug can result in multiple tasks waiting on rcu_migrate_wq: there can be one CPU-hotplug notifier, and, because rcu_barrier() invokes wait_migrated_callbacks() outside of rcu_barrier_mutex, an arbitrarily large number of tasks executing in _rcu_barrier(). This situation could result in hangs, because rcu_migrate_callback() would wake up but one task.

This patch addresses this problem by awakening all sleepers.

Located-by Mathieu Desnoyers <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

---

 rcupdate.c |    9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

kernel/rcupdate.c +5 -0

index bd5d5c8..6c59e1b 100644
@@ -211,10 +211,17 @@ void rcu_barrier_sched(void)
 }
 EXPORT_SYMBOL_GPL(rcu_barrier_sched);
 
+/*
+ * Wake up anyone waiting on migration of RCU callbacks from a CPU
+ * going offline.  There can be only one CPU-hotplug notifier waiting,
+ * but there can be any number of rcu_barrier() invocations waiting,
+ * due to the fact that rcu_barrier() does its wait outside of the

kernel/rcupdate.c void rcu_barrier_sched(void) · line 227

Quoted 10 lines

kernel/rcupdate.c +3 -1

+ * rcu_barrier_mutex.
+ */
 static void rcu_migrate_callback(struct rcu_head *notused)
 {
 	if (atomic_dec_and_test(&rcu_migrate_type_count))
-		wake_up(&rcu_migrate_wq);
+		wake_up_all(&rcu_migrate_wq);
 }
 
 extern int rcu_cpu_notify(struct notifier_block *self,

Maybe wake_up_all() is good for readability. But wake_up() is enough for the logic.

In wait_migrated_callbacks(), we use wait_event(), The current thread is queued as a non-exclusive waiter. And wake_up() wakes up all non-exclusive waiter. (We only have non-exclusive waiters in the rcu_migrate_wq)