Skip to contentSkip to conversation

No-CBs bypass addition for v5.4

PATCH In review Paul E. McKenney 5 participants 57 messages Aug 2, 2019 – Aug 15, 2019

Subsystems (4)
lore
Share & tools
lore
#18 lore raw

RFC PATCH 7/14 rcu/nocb: Reduce nocb_cb_wait() leaf rcu_node ->lock contention

In reply to Paul E. McKenney

Currently, nocb_cb_wait() advances callbacks on each pass through its loop, though only if it succeeds in conditionally acquiring its leaf rcu_node structure's ->lock. Despite the conditional acquisition of ->lock, this does increase contention. This commit therefore avoids advancing callbacks unless there are callbacks in ->cblist whose grace period has completed.

Note that nocb_cb_wait() doesn't worry about callbacks that have not yet been assigned a grace period. The idea is that the only reason for nocb_cb_wait() to advance callbacks is to allow it to continue invoking callbacks. Time will tell whether this is the correct choice.

---
 kernel/rcu/tree_plugin.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +4 -1
index 5db39ce1cae1..02739366ef5d 100644
@@ -2076,6 +2076,7 @@ static int rcu_nocb_gp_kthread(void *arg)
  */
 static void nocb_cb_wait(struct rcu_data *rdp)
 {
+	unsigned long cur_gp_seq;
 	unsigned long flags;
 	bool needwake_gp = false;
 	struct rcu_node *rnp = rdp->mynode;
@@ -2088,7 +2089,9 @@ static void nocb_cb_wait(struct rcu_data *rdp)
 	local_bh_enable();
 	lockdep_assert_irqs_enabled();
 	rcu_nocb_lock_irqsave(rdp, flags);
-	if (raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
+	if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
+	    rcu_seq_done(&rnp->gp_seq, cur_gp_seq) &&
+	    raw_spin_trylock_rcu_node(rnp)) { /* irqs already disabled. */
 		needwake_gp = rcu_advance_cbs(rdp->mynode, rdp);
 		raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */
 	}
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH 1/14 rcu/nocb: Atomic ->len field in rcu_segcblist structure

In reply to Paul E. McKenney

Upcoming ->nocb_lock contention-reduction work requires that the rcu_segcblist structure's ->len field be concurrently manipulated, but only if there are no-CBs CPUs in the kernel. This commit therefore makes this ->len field be an atomic_long_t, but only in CONFIG_RCU_NOCB_CPU=y kernels.

---
 include/linux/rcu_segcblist.h |  4 ++
 kernel/rcu/rcu_segcblist.c    | 86 ++++++++++++++++++++++++++++++++---
 kernel/rcu/rcu_segcblist.h    | 12 ++++-
 3 files changed, 94 insertions(+), 8 deletions(-)
include/linux/rcu_segcblist.h +4 -0
index 82977726da29..38e4b811af9b 100644
@@ -65,7 +65,11 @@ struct rcu_segcblist {
 	struct rcu_head *head;
 	struct rcu_head **tails[RCU_CBLIST_NSEGS];
 	unsigned long gp_seq[RCU_CBLIST_NSEGS];
+#ifdef CONFIG_RCU_NOCB_CPU
+	atomic_long_t len;
+#else
 	long len;
+#endif
 	long len_lazy;
 	u8 enabled;
 	u8 offloaded;
kernel/rcu/rcu_segcblist.c +79 -7

129 lines not on this page · raw message

kernel/rcu/rcu_segcblist.h +11 -1
index db38f0a512c4..1ff996647d3c 100644
@@ -9,6 +9,12 @@
 
 #include <linux/rcu_segcblist.h>
 
+/* Return number of callbacks in the specified callback list. */
+static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp)
+{
+	return READ_ONCE(rclp->len);
+}
+
 /*
  * Account for the fact that a previously dequeued callback turned out
  * to be marked as lazy.
@@ -42,7 +48,11 @@ static inline bool rcu_segcblist_empty(struct rcu_segcblist *rsclp)
 /* Return number of callbacks in segmented callback list. */
 static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)
 {
+#ifdef CONFIG_RCU_NOCB_CPU
+	return atomic_long_read(&rsclp->len);
+#else
 	return READ_ONCE(rsclp->len);
+#endif
 }
 
 /* Return number of lazy callbacks in segmented callback list. */
@@ -54,7 +64,7 @@ static inline long rcu_segcblist_n_lazy_cbs(struct rcu_segcblist *rsclp)
 /* Return number of lazy callbacks in segmented callback list. */
 static inline long rcu_segcblist_n_nonlazy_cbs(struct rcu_segcblist *rsclp)
 {
-	return rsclp->len - rsclp->len_lazy;
+	return rcu_segcblist_n_cbs(rsclp) - rsclp->len_lazy;
 }
 
 /*
Signature · 1 line
2.17.1
#24 lore raw

RFC PATCH 13/14 rcutorture: Force on tick for readers and callback flooders

In reply to Paul E. McKenney

Readers and callback flooders in the rcutorture stress-test suite run for extended time periods by design. They do take pains to relinquish the CPU from time to time, but in some cases this relies on the scheduler being active, which in turn relies on the scheduler-clock interrupt firing from time to time.

This commit therefore forces scheduling-clock interrupts within these loops. While in the area, this commit also prevents rcu_torture_reader()'s occasional timed sleeps from delaying shutdown.

---
 kernel/rcu/rcutorture.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
kernel/rcu/rcutorture.c +18 -6
index 3c9feca1eab1..bf08aa783ecc 100644
@@ -44,6 +44,7 @@
 #include <linux/sched/debug.h>
 #include <linux/sched/sysctl.h>
 #include <linux/oom.h>
+#include <linux/tick.h>
 
 #include "rcu.h"
 
@@ -1363,15 +1364,16 @@ rcu_torture_reader(void *arg)
 	set_user_nice(current, MAX_NICE);
 	if (irqreader && cur_ops->irq_capable)
 		timer_setup_on_stack(&t, rcu_torture_timer, 0);
-
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_set_task(current, TICK_DEP_MASK_RCU);
 	do {
 		if (irqreader && cur_ops->irq_capable) {
 			if (!timer_pending(&t))
 				mod_timer(&t, jiffies + 1);
 		}
-		if (!rcu_torture_one_read(&rand))
+		if (!rcu_torture_one_read(&rand) && !torture_must_stop())
 			schedule_timeout_interruptible(HZ);
-		if (time_after(jiffies, lastsleep)) {
+		if (time_after(jiffies, lastsleep) && !torture_must_stop()) {
 			schedule_timeout_interruptible(1);
 			lastsleep = jiffies + 10;
 		}
@@ -1383,6 +1385,8 @@ rcu_torture_reader(void *arg)
 		del_timer_sync(&t);
 		destroy_timer_on_stack(&t);
 	}
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_clear_task(current, TICK_DEP_MASK_RCU);
 	torture_kthread_stopping("rcu_torture_reader");
 	return 0;
 }
@@ -1729,10 +1733,10 @@ static void rcu_torture_fwd_prog_cond_resched(unsigned long iter)
 		// Real call_rcu() floods hit userspace, so emulate that.
 		if (need_resched() || (iter & 0xfff))
 			schedule();
-	} else {
-		// No userspace emulation: CB invocation throttles call_rcu()
-		cond_resched();
+		return;
 	}
+	// No userspace emulation: CB invocation throttles call_rcu()
+	cond_resched();
 }
 
 /*
@@ -1781,6 +1785,8 @@ static void rcu_torture_fwd_prog_nr(int *tested, int *tested_tries)
 		init_rcu_head_on_stack(&fcs.rh);
 		selfpropcb = true;
 	}
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_set_task(current, TICK_DEP_MASK_RCU);
 
 	/* Tight loop containing cond_resched(). */
 	WRITE_ONCE(rcu_fwd_cb_nodelay, true);
@@ -1826,6 +1832,8 @@ static void rcu_torture_fwd_prog_nr(int *tested, int *tested_tries)
 		destroy_rcu_head_on_stack(&fcs.rh);
 	}
 	schedule_timeout_uninterruptible(HZ / 10); /* Let kthreads recover. */
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_clear_task(current, TICK_DEP_MASK_RCU);
 	WRITE_ONCE(rcu_fwd_cb_nodelay, false);
 }
 
@@ -1865,6 +1873,8 @@ static void rcu_torture_fwd_prog_cr(void)
 	cver = READ_ONCE(rcu_torture_current_version);
 	gps = cur_ops->get_gp_seq();
 	rcu_launder_gp_seq_start = gps;
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_set_task(current, TICK_DEP_MASK_RCU);
 	while (time_before(jiffies, stopat) &&
 	       !shutdown_time_arrived() &&
 	       !READ_ONCE(rcu_fwd_emergency_stop) && !torture_must_stop()) {
@@ -1911,6 +1921,8 @@ static void rcu_torture_fwd_prog_cr(void)
 		rcu_torture_fwd_cb_hist();
 	}
 	schedule_timeout_uninterruptible(HZ); /* Let CBs drain. */
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_clear_task(current, TICK_DEP_MASK_RCU);
 	WRITE_ONCE(rcu_fwd_cb_nodelay, false);
 }
 
Signature · 1 line
2.17.1
#0 lore raw

cover No-CBs bypass addition for v5.4

Hello!

This series is a sneak preview of additional work for the move of no-CBs CPUs to the ->cblist segmented RCU callback list. This work adds a ->nocb_bypass list with its own lock to further reduce contention. This series also includes some nascent work to turn the scheduling-clock interrupt back on for nohz_full CPUs doing heavy rcutorture work or RCU callback invocation, both of which can remain in the kernel for long time periods, which in turn can impede CPU hotplug removals. (On some systems "impede" means up to seven minutes for stop-machine to actually get things to stop, a problem that has not yet been observed on no-CBs CPUs that are not also nohz_full CPUs.)

1. Atomic ->len field in rcu_segcblist structure.

2. Add bypass callback queueing in ->nocb_bypass with its own

	->nocb_bypass_lock.

3. (Experimental) Check use and usefulness of ->nocb_lock_contended.

4. Print no-CBs diagnostics when rcutorture writer unduly delayed.

5. Avoid synchronous wakeup in __call_rcu_nocb_wake().

6. Advance CBs after merge in rcutree_migrate_callbacks() to
avoid unnecessary invocation delays.

7. Reduce nocb_cb_wait() leaf rcu_node ->lock contention.

8. Reduce __call_rcu_nocb_wake() leaf rcu_node ->lock contention.

9. Don't wake no-CBs GP kthread if timer posted under overload,
thus reducing overhead in the overload case.

10. Allow rcu_do_batch() to dynamically adjust batch sizes, courtesy
of Eric Dumazet.

11. (Experimental) Add TICK_DEP_BIT_RCU, courtesy of Frederic Weisbecker.

12. Force on tick when invoking lots of callbacks to reduce the
probability of long stop-machine delays.

13. Force on tick for readers and callback flooders, again to reduce
the probability of long stop-machine delays.

14. (Experimental and likely quite imperfect) Make multi_cpu_stop()
enable tick on all online CPUs, yet again to reduce the probability of long stop-machine delays.

Thanx, Paul

------------------------------------------------------------------------

 include/linux/rcu_segcblist.h |    4 
 include/linux/tick.h          |    7 
 kernel/rcu/rcu_segcblist.c    |  116 +++++++++-
 kernel/rcu/rcu_segcblist.h    |   17 +
 kernel/rcu/rcutorture.c       |   25 +-
 kernel/rcu/tree.c             |   41 +++
 kernel/rcu/tree.h             |   35 ++-
 kernel/rcu/tree_plugin.h      |  486 +++++++++++++++++++++++++++++++++++++-----
 kernel/rcu/tree_stall.h       |    5 
 kernel/stop_machine.c         |    9 
 kernel/time/tick-sched.c      |    2 
 11 files changed, 667 insertions(+), 80 deletions(-)
#23 lore raw

RFC PATCH 12/14 rcu/nohz: Force on tick when invoking lots of callbacks

Callback invocation can run for a significant time period, and within CONFIG_NO_HZ_FULL=y kernels, this period will be devoid of scheduler-clock interrupts. In-kernel execution without such interrupts can cause all manner of malfunction, with RCU CPU stall warnings being but one result.

This commit therefore forces scheduling-clock interrupts on whenever more than a few RCU callbacks are invoked. Because offloaded callback invocation can be preempted, this forcing is withdrawn on each context switch. This in turn requires that the loop invoking RCU callbacks reiterate the forcing periodically.

---
 kernel/rcu/tree.c | 4 ++++
 1 file changed, 4 insertions(+)
kernel/rcu/tree.c +4 -0
index 71395e91b876..b80eb11c19e1 100644
@@ -2151,6 +2151,8 @@ static void rcu_do_batch(struct rcu_data *rdp)
 	rcu_nocb_unlock_irqrestore(rdp, flags);
 
 	/* Invoke callbacks. */
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_set_task(current, TICK_DEP_MASK_RCU);
 	rhp = rcu_cblist_dequeue(&rcl);
 	for (; rhp; rhp = rcu_cblist_dequeue(&rcl)) {
 		debug_rcu_head_unqueue(rhp);
@@ -2217,6 +2219,8 @@ static void rcu_do_batch(struct rcu_data *rdp)
 	/* Re-invoke RCU core processing if there are callbacks remaining. */
 	if (!offloaded && rcu_segcblist_ready_cbs(&rdp->cblist))
 		invoke_rcu_core();
+	if (IS_ENABLED(CONFIG_NO_HZ_FULL))
+		tick_dep_clear_task(current, TICK_DEP_MASK_RCU);
 }
 
 /*
Signature · 1 line
2.17.1
#25 lore raw

RFC PATCH 14/14 rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

---
 kernel/stop_machine.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/stop_machine.c +8 -1
index b4f83f7bdf86..a2659f61ed92 100644
@@ -20,6 +20,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/tick.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
 {
 	struct multi_stop_data *msdata = data;
 	enum multi_stop_state curstate = MULTI_STOP_NONE;
-	int cpu = smp_processor_id(), err = 0;
+	int cpu, err = 0;
 	const struct cpumask *cpumask;
 	unsigned long flags;
 	bool is_active;
 
+	for_each_online_cpu(cpu)
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);
+
 	/*
 	 * When called from stop_machine_from_inactive_cpu(), irq might
 	 * already be disabled.  Save the state and restore it on exit.
 	 */
+	cpu = smp_processor_id();
 	local_save_flags(flags);
 
 	if (!msdata->active_cpus) {
@@ -236,6 +241,8 @@ static int multi_cpu_stop(void *data)
 	} while (curstate != MULTI_STOP_EXIT);
 
 	local_irq_restore(flags);
+	for_each_online_cpu(cpu)
+		tick_nohz_dep_clear_cpu(cpu, TICK_DEP_MASK_RCU);
 	return err;
 }
 
Signature · 1 line
2.17.1
#17 lore raw

RFC PATCH 6/14 rcu/nocb: Advance CBs after merge in rcutree_migrate_callbacks()

In reply to Paul E. McKenney

The rcutree_migrate_callbacks() invokes rcu_advance_cbs() on both the offlined CPU's ->cblist and that of the surviving CPU, then merges them. However, after the merge, and of the offlined CPU's callbacks that were not ready to be invoked will no longer be associated with a grace-period number. This commit therefore invokes rcu_advance_cbs() one more time on the merged ->cblist in order to assign a grace-period number to these callbacks.

---
 kernel/rcu/tree.c | 1 +
 1 file changed, 1 insertion(+)
kernel/rcu/tree.c +1 -0
index 457623100d12..3e89b5b83ea0 100644
@@ -3205,6 +3205,7 @@ void rcutree_migrate_callbacks(int cpu)
 	needwake = rcu_advance_cbs(my_rnp, rdp) ||
 		   rcu_advance_cbs(my_rnp, my_rdp);
 	rcu_segcblist_merge(&my_rdp->cblist, &rdp->cblist);
+	needwake = needwake || rcu_advance_cbs(my_rnp, my_rdp);
 	rcu_segcblist_disable(&rdp->cblist);
 	WARN_ON_ONCE(rcu_segcblist_empty(&my_rdp->cblist) !=
 		     !rcu_segcblist_n_cbs(&my_rdp->cblist));
Signature · 1 line
2.17.1
#14 lore raw

RFC PATCH 3/14 rcu/nocb: EXP Check use and usefulness of ->nocb_lock_contended

In reply to Paul E. McKenney
---
 kernel/rcu/tree_plugin.h | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +3 -1
index bb906295538d..fcbd61738dff 100644
@@ -1513,6 +1513,7 @@ static void rcu_nocb_bypass_lock(struct rcu_data *rdp)
 	if (raw_spin_trylock(&rdp->nocb_bypass_lock))
 		return;
 	atomic_inc(&rdp->nocb_lock_contended);
+	WARN_ON_ONCE(smp_processor_id() != rdp->cpu);
 	smp_mb__after_atomic(); /* atomic_inc() before lock. */
 	raw_spin_lock(&rdp->nocb_bypass_lock);
 	smp_mb__before_atomic(); /* atomic_dec() after lock. */
@@ -1531,7 +1532,8 @@ static void rcu_nocb_bypass_lock(struct rcu_data *rdp)
  */
 static void rcu_nocb_wait_contended(struct rcu_data *rdp)
 {
-	while (atomic_read(&rdp->nocb_lock_contended))
+	WARN_ON_ONCE(smp_processor_id() != rdp->cpu);
+	while (WARN_ON_ONCE(atomic_read(&rdp->nocb_lock_contended)))
 		cpu_relax();
 }
 
Signature · 1 line
2.17.1
#22 lore raw

RFC PATCH 11/14 EXP nohz: Add TICK_DEP_BIT_RCU

In reply to Paul E. McKenney

From: Frederic Weisbecker <[email protected]>

Indeed, kernel code is assumed to be quick enough (between two extended grace periods) to avoid running the tick for RCU. But some long lasting kernel code may require to tick temporarily.

You can use tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU) with the
following: @@@

[ paulmck: Enable use within rcutorture and from !NO_HZ_FULL kernels. ]
---
 include/linux/tick.h     | 7 ++++++-
 kernel/time/tick-sched.c | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)
include/linux/tick.h +6 -1
index 196a0a7bfc4f..0dea6fb33a11 100644
@@ -108,7 +108,8 @@ enum tick_dep_bits {
 	TICK_DEP_BIT_POSIX_TIMER	= 0,
 	TICK_DEP_BIT_PERF_EVENTS	= 1,
 	TICK_DEP_BIT_SCHED		= 2,
-	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3
+	TICK_DEP_BIT_CLOCK_UNSTABLE	= 3,
+	TICK_DEP_BIT_RCU		= 4
 };
 
 #define TICK_DEP_MASK_NONE		0
@@ -116,6 +117,7 @@ enum tick_dep_bits {
 #define TICK_DEP_MASK_PERF_EVENTS	(1 << TICK_DEP_BIT_PERF_EVENTS)
 #define TICK_DEP_MASK_SCHED		(1 << TICK_DEP_BIT_SCHED)
 #define TICK_DEP_MASK_CLOCK_UNSTABLE	(1 << TICK_DEP_BIT_CLOCK_UNSTABLE)
+#define TICK_DEP_MASK_RCU		(1 << TICK_DEP_BIT_RCU)
 
 #ifdef CONFIG_NO_HZ_COMMON
 extern bool tick_nohz_enabled;
@@ -258,6 +260,9 @@ static inline bool tick_nohz_full_enabled(void) { return false; }
 static inline bool tick_nohz_full_cpu(int cpu) { return false; }
 static inline void tick_nohz_full_add_cpus_to(struct cpumask *mask) { }
 
+static inline void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
+static inline void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit) { }
+
 static inline void tick_dep_set(enum tick_dep_bits bit) { }
 static inline void tick_dep_clear(enum tick_dep_bits bit) { }
 static inline void tick_dep_set_cpu(int cpu, enum tick_dep_bits bit) { }
kernel/time/tick-sched.c +2 -0
index be9707f68024..5d4844d543cf 100644
@@ -324,6 +324,7 @@ void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit)
 		preempt_enable();
 	}
 }
+EXPORT_SYMBOL_GPL(tick_nohz_dep_set_cpu);
 
 void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 {
@@ -331,6 +332,7 @@ void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 
 	atomic_andnot(BIT(bit), &ts->tick_dep_mask);
 }
+EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
 /*
  * Set a per-task tick dependency. Posix CPU timers need this in order to elapse
Signature · 1 line
2.17.1
#16 lore raw

RFC PATCH 5/14 rcu/nocb: Avoid synchronous wakeup in __call_rcu_nocb_wake()

In reply to Paul E. McKenney

When callbacks are in full flow, the common case is waiting for a grace period, and this grace period will normally take a few jiffies to complete. It therefore isn't all that helpful for __call_rcu_nocb_wake() to do a synchronous wakeup in this case. This commit therefore turns this into a timer-based deferred wakeup of the no-CBs grace-period kthread.

---
 kernel/rcu/tree_plugin.h | 19 +++++--------------
 1 file changed, 5 insertions(+), 14 deletions(-)
kernel/rcu/tree_plugin.h +5 -14
index aaeb8a658f4b..5db39ce1cae1 100644
@@ -1897,22 +1897,13 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 	} else if (len > rdp->qlen_last_fqs_check + qhimark) {
 		/* ... or if many callbacks queued. */
 		rdp->qlen_last_fqs_check = len;
-		if (!rdp->nocb_cb_sleep &&
-		    rcu_segcblist_ready_cbs(&rdp->cblist)) {
-			// Already going full tilt, so don't try to rewake.
-			rcu_nocb_unlock_irqrestore(rdp, flags);
-		} else {
+		if (rdp->nocb_cb_sleep ||
+		    !rcu_segcblist_ready_cbs(&rdp->cblist)) {
 			rcu_advance_cbs_nowake(rdp->mynode, rdp);
-			if (!irqs_disabled_flags(flags)) {
-				wake_nocb_gp(rdp, false, flags);
-				trace_rcu_nocb_wake(rcu_state.name, rdp->cpu,
-						    TPS("WakeOvf"));
-			} else {
-				wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE,
-						   TPS("WakeOvfIsDeferred"));
-				rcu_nocb_unlock_irqrestore(rdp, flags);
-			}
+			wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE,
+					   TPS("WakeOvfIsDeferred"));
 		}
+		rcu_nocb_unlock_irqrestore(rdp, flags);
 	} else {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot"));
 		rcu_nocb_unlock_irqrestore(rdp, flags);
Signature · 1 line
2.17.1
#15 lore raw

RFC PATCH 4/14 rcu/nocb: Print no-CBs diagnostics when rcutorture writer unduly delayed

In reply to Paul E. McKenney

This commit causes locking, sleeping, and callback state to be printed for no-CBs CPUs when the rcutorture writer is delayed sufficiently for rcutorture to complain.

---
 kernel/rcu/rcutorture.c  |  1 +
 kernel/rcu/tree.h        |  7 +++-
 kernel/rcu/tree_plugin.h | 82 ++++++++++++++++++++++++++++++++++++++++
 kernel/rcu/tree_stall.h  |  5 +++
 4 files changed, 94 insertions(+), 1 deletion(-)
kernel/rcu/rcutorture.c +1 -0
index b22947324423..3c9feca1eab1 100644
@@ -2176,6 +2176,7 @@ rcu_torture_cleanup(void)
 		return;
 	}
 
+	show_rcu_gp_kthreads();
 	rcu_torture_barrier_cleanup();
 	torture_stop_kthread(rcu_torture_fwd_prog, fwd_prog_task);
 	torture_stop_kthread(rcu_torture_stall, stall_task);
kernel/rcu/tree.h +6 -1
index e4df86db8137..c612f306fe89 100644
@@ -212,7 +212,11 @@ struct rcu_data {
 	/* The following fields are used by GP kthread, hence own cacheline. */
 	raw_spinlock_t nocb_gp_lock ____cacheline_internodealigned_in_smp;
 	struct timer_list nocb_bypass_timer; /* Force nocb_bypass flush. */
-	bool nocb_gp_sleep;		/* Is the nocb GP thread asleep? */
+	u8 nocb_gp_sleep;		/* Is the nocb GP thread asleep? */
+	u8 nocb_gp_bypass;		/* Found a bypass on last scan? */
+	u8 nocb_gp_gp;			/* GP to wait for on last scan? */
+	unsigned long nocb_gp_seq;	/*  If so, ->gp_seq to wait for. */
+	unsigned long nocb_gp_loops;	/* # passes through wait code. */
 	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */
 	bool nocb_cb_sleep;		/* Is the nocb CB thread asleep? */
 	struct task_struct *nocb_cb_kthread;
@@ -438,6 +442,7 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp);
 static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp);
 static void rcu_spawn_cpu_nocb_kthread(int cpu);
 static void __init rcu_spawn_nocb_kthreads(void);
+static void show_rcu_nocb_state(struct rcu_data *rdp);
 static void rcu_nocb_lock(struct rcu_data *rdp);
 static void rcu_nocb_unlock(struct rcu_data *rdp);
 static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
kernel/rcu/tree_plugin.h +82 -0

112 lines not on this page · raw message

kernel/rcu/tree_stall.h +5 -0
index 0627a66699a6..841ab43f3e60 100644
@@ -589,6 +589,11 @@ void show_rcu_gp_kthreads(void)
 				cpu, (long)rdp->gp_seq_needed);
 		}
 	}
+	for_each_possible_cpu(cpu) {
+		rdp = per_cpu_ptr(&rcu_data, cpu);
+		if (rcu_segcblist_is_offloaded(&rdp->cblist))
+			show_rcu_nocb_state(rdp);
+	}
 	/* sched_show_task(rcu_state.gp_kthread); */
 }
 EXPORT_SYMBOL_GPL(show_rcu_gp_kthreads);
Signature · 1 line
2.17.1
#21 lore raw

RFC PATCH 10/14 rcu: Allow rcu_do_batch() to dynamically adjust batch sizes

In reply to Paul E. McKenney

From: Eric Dumazet <[email protected]>

Bimodal behavior of rcu_do_batch() is not really suited to Google applications like gfe servers.

When a process with millions of sockets exits, closing all files queues two rcu callbacks per socket.

This eventually reaches the point where RCU enters an emergency mode, where rcu_do_batch() do not return until whole queue is flushed.

Each rcu callback lasts at least 70 nsec, so with millions of elements, we easily spend more than 100 msec without rescheduling.

Goal of this patch is to avoid the infamous message like following "need_resched set for > 51999388 ns (52 ticks) without schedule"

We dynamically adjust the number of elements we process, instead of 10 / INFINITE choices, we use a floor of ~1 % of current entries.

If the number is above 1000, we switch to a time based limit of 3 msec per batch, adjustable with /sys/module/rcutree/parameters/rcu_resched_ns

[ paulmck: Forward-port and remove debug statements. ]
---
 kernel/rcu/tree.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +19 -1
index 3e89b5b83ea0..71395e91b876 100644
@@ -56,6 +56,7 @@
 #include <linux/smpboot.h>
 #include <linux/jiffies.h>
 #include <linux/sched/isolation.h>
+#include <linux/sched/clock.h>
 #include "../time/tick-internal.h"
 
 #include "tree.h"
@@ -416,6 +417,12 @@ module_param(qlowmark, long, 0444);
 static ulong jiffies_till_first_fqs = ULONG_MAX;
 static ulong jiffies_till_next_fqs = ULONG_MAX;
 static bool rcu_kick_kthreads;
+static int rcu_divisor = 7;
+module_param(rcu_divisor, int, 0644);
+
+/* Force an exit from rcu_do_batch() after 3 milliseconds. */
+static long rcu_resched_ns = 3 * NSEC_PER_MSEC;
+module_param(rcu_resched_ns, long, 0644);
 
 /*
  * How long the grace period must be before we start recruiting
@@ -2109,6 +2116,7 @@ static void rcu_do_batch(struct rcu_data *rdp)
 	struct rcu_head *rhp;
 	struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
 	long bl, count;
+	long pending, tlimit = 0;
 
 	/* If no callbacks are ready, just return. */
 	if (!rcu_segcblist_ready_cbs(&rdp->cblist)) {
@@ -2130,7 +2138,10 @@ static void rcu_do_batch(struct rcu_data *rdp)
 	local_irq_save(flags);
 	rcu_nocb_lock(rdp);
 	WARN_ON_ONCE(cpu_is_offline(smp_processor_id()));
-	bl = rdp->blimit;
+	pending = rcu_segcblist_n_cbs(&rdp->cblist);
+	bl = max(rdp->blimit, pending >> rcu_divisor);
+	if (unlikely(bl > 100))
+		tlimit = local_clock() + rcu_resched_ns;
 	trace_rcu_batch_start(rcu_state.name,
 			      rcu_segcblist_n_lazy_cbs(&rdp->cblist),
 			      rcu_segcblist_n_cbs(&rdp->cblist), bl);
@@ -2153,6 +2164,13 @@ static void rcu_do_batch(struct rcu_data *rdp)
 		    (need_resched() ||
 		     (!is_idle_task(current) && !rcu_is_callbacks_kthread())))
 			break;
+		if (unlikely(tlimit)) {
+			/* only call local_clock() every 32 callbacks */
+			if (likely((-rcl.len & 31) || local_clock() < tlimit))
+				continue;
+			/* Exceeded the time limit, so leave. */
+			break;
+		}
 		if (offloaded) {
 			WARN_ON_ONCE(in_serving_softirq());
 			local_bh_enable();
Signature · 1 line
2.17.1
#20 lore raw

RFC PATCH 9/14 rcu/nocb: Don't wake no-CBs GP kthread if timer posted under overload

In reply to Paul E. McKenney

When under overload conditions, __call_rcu_nocb_wake() will wake the no-CBs GP kthread any time the no-CBs CB kthread is asleep or there are no ready-to-invoke callbacks, but only after a timer delay. If the no-CBs GP kthread has a ->nocb_bypass_timer pending, the deferred wakeup from __call_rcu_nocb_wake() is redundant. This commit therefore makes __call_rcu_nocb_wake() avoid posting the redundant deferred wakeup if ->nocb_bypass_timer is pending. This requires adding a bit of ordering of timer actions.

---
 kernel/rcu/tree_plugin.h | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +5 -2
index efd7f6fa2790..379cb7e50a62 100644
@@ -1906,8 +1906,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 			rcu_advance_cbs_nowake(rdp->mynode, rdp);
 			rdp->nocb_gp_adv_time = j;
 		}
-		if (rdp->nocb_cb_sleep ||
-		    !rcu_segcblist_ready_cbs(&rdp->cblist))
+		smp_mb(); /* Enqueue before timer_pending(). */
+		if ((rdp->nocb_cb_sleep ||
+		     !rcu_segcblist_ready_cbs(&rdp->cblist)) &&
+		    !timer_pending(&rdp->nocb_bypass_timer))
 			wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE,
 					   TPS("WakeOvfIsDeferred"));
 		rcu_nocb_unlock_irqrestore(rdp, flags);
@@ -1926,6 +1928,7 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)
 
 	trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Timer"));
 	rcu_nocb_lock_irqsave(rdp, flags);
+	smp_mb__after_spinlock(); /* Timer expire before wakeup. */
 	__call_rcu_nocb_wake(rdp, true, flags);
 }
 
Signature · 1 line
2.17.1
#6 lore raw

RFC PATCH 2/14 rcu/nocb: Add bypass callback queueing

In reply to Paul E. McKenney

Use of the rcu_data structure's segmented ->cblist for no-CBs CPUs takes advantage of unrelated grace periods, thus reducing the memory footprint in the face of floods of call_rcu() invocations. However, the ->cblist field is a more-complex rcu_segcblist structure which must be protected via locking. Even though there are only three entities which can acquire this lock (the CPU invoking call_rcu(), the no-CBs grace-period kthread, and the no-CBs callbacks kthread), the contention on this lock is excessive under heavy stress.

This commit therefore greatly reduces contention by provisioning an rcu_cblist structure field named ->nocb_bypass within the rcu_data structure. Each no-CBs CPU is permitted only a limited number of enqueues onto the ->cblist per jiffy, controlled by a new nocb_nobypass_lim_per_jiffy kernel boot parameter that defaults to about 16 enqueues per millisecond (16 * 1000 / HZ). When that limit is exceeded, the CPU instead enqueues onto the new ->nocb_bypass.

The ->nocb_bypass is flushed into the ->cblist every jiffy or when the number of callbacks on ->nocb_bypass exceeds qhimark, whichever happens first. During call_rcu() floods, this flushing is carried out by the CPU during the course of its call_rcu() invocations. However, a CPU could simply stop invoking call_rcu() at any time. The no-CBs grace-period kthread therefore carries out less-aggressive flushing (every few jiffies or when the number of callbacks on ->nocb_bypass exceeds (2 * qhimark), whichever comes first). This means that the no-CBs grace-period kthread cannot be permitted to do unbounded waits while there are callbacks on ->nocb_bypass. A ->nocb_bypass_timer is used to provide the needed wakeups.

---
 kernel/rcu/rcu_segcblist.c |  30 ++++
 kernel/rcu/rcu_segcblist.h |   5 +
 kernel/rcu/tree.c          |  16 +-
 kernel/rcu/tree.h          |  28 +--
 kernel/rcu/tree_plugin.h   | 356 ++++++++++++++++++++++++++++++++++---
 5 files changed, 394 insertions(+), 41 deletions(-)
kernel/rcu/rcu_segcblist.c +30 -0
index ff431cc83037..495c58ce1640 100644
@@ -36,6 +36,36 @@ void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp)
 	WRITE_ONCE(rclp->len, rclp->len + 1);
 }
 
+/*
+ * Flush the second rcu_cblist structure onto the first one, obliterating
+ * any contents of the first.  If rhp is non-NULL, enqueue it as the sole
+ * element of the second rcu_cblist structure, but ensuring that the second
+ * rcu_cblist structure, if initially non-empty, always appears non-empty
+ * throughout the process.  If rdp is NULL, the second rcu_cblist structure
+ * is instead initialized to empty.
+ */
+void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,
+			      struct rcu_cblist *srclp,
+			      struct rcu_head *rhp)
+{
+	drclp->head = srclp->head;
+	if (drclp->head)
+		drclp->tail = srclp->tail;
+	else
+		drclp->tail = &drclp->head;
+	drclp->len = srclp->len;
+	drclp->len_lazy = srclp->len_lazy;
+	if (!rhp) {
+		rcu_cblist_init(srclp);
+	} else {
+		rhp->next = NULL;
+		srclp->head = rhp;
+		srclp->tail = &rhp->next;
+		WRITE_ONCE(srclp->len, 1);
+		srclp->len_lazy = 0;
+	}
+}
+
 /*
  * Dequeue the oldest rcu_head structure from the specified callback
  * list.  This function assumes that the callback is non-lazy, but
kernel/rcu/rcu_segcblist.h +5 -0
index 1ff996647d3c..815c2fdd3fcc 100644
@@ -25,6 +25,10 @@ static inline void rcu_cblist_dequeued_lazy(struct rcu_cblist *rclp)
 }
 
 void rcu_cblist_init(struct rcu_cblist *rclp);
+void rcu_cblist_enqueue(struct rcu_cblist *rclp, struct rcu_head *rhp);
+void rcu_cblist_flush_enqueue(struct rcu_cblist *drclp,
+			      struct rcu_cblist *srclp,
+			      struct rcu_head *rhp);
 struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp);
 
 /*
@@ -92,6 +96,7 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)
 	return !READ_ONCE(*READ_ONCE(rsclp->tails[seg]));
 }
 
+void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);
 void rcu_segcblist_init(struct rcu_segcblist *rsclp);
 void rcu_segcblist_disable(struct rcu_segcblist *rsclp);
 void rcu_segcblist_offload(struct rcu_segcblist *rsclp);
kernel/rcu/tree.c +12 -4
index ec320658aeef..457623100d12 100644
@@ -1251,6 +1251,7 @@ static bool rcu_accelerate_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
 	unsigned long gp_seq_req;
 	bool ret = false;
 
+	rcu_lockdep_assert_cblist_protected(rdp);
 	raw_lockdep_assert_held_rcu_node(rnp);
 
 	/* If no pending (not yet ready to invoke) callbacks, nothing to do. */
@@ -1292,7 +1293,7 @@ static void rcu_accelerate_cbs_unlocked(struct rcu_node *rnp,
 	unsigned long c;
 	bool needwake;
 
-	lockdep_assert_irqs_disabled();
+	rcu_lockdep_assert_cblist_protected(rdp);
 	c = rcu_seq_snap(&rcu_state.gp_seq);
 	if (!rdp->gpwrap && ULONG_CMP_GE(rdp->gp_seq_needed, c)) {
 		/* Old request still live, so mark recent callbacks. */
@@ -1318,6 +1319,7 @@ static void rcu_accelerate_cbs_unlocked(struct rcu_node *rnp,
  */
 static bool rcu_advance_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
 {
+	rcu_lockdep_assert_cblist_protected(rdp);
 	raw_lockdep_assert_held_rcu_node(rnp);
 
 	/* If no pending (not yet ready to invoke) callbacks, nothing to do. */
@@ -1341,6 +1343,7 @@ static bool rcu_advance_cbs(struct rcu_node *rnp, struct rcu_data *rdp)
 static void __maybe_unused rcu_advance_cbs_nowake(struct rcu_node *rnp,
 						  struct rcu_data *rdp)
 {
+	rcu_lockdep_assert_cblist_protected(rdp);
 	if (!rcu_seq_state(rcu_seq_current(&rnp->gp_seq)) ||
 	    !raw_spin_trylock_rcu_node(rnp))
 		return;
@@ -2187,7 +2190,9 @@ static void rcu_do_batch(struct rcu_data *rdp)
 	 * The following usually indicates a double call_rcu().  To track
 	 * this down, try building with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y.
 	 */
-	WARN_ON_ONCE(rcu_segcblist_empty(&rdp->cblist) != (count == 0));
+	WARN_ON_ONCE(count == 0 && !rcu_segcblist_empty(&rdp->cblist));
+	WARN_ON_ONCE(!IS_ENABLED(CONFIG_RCU_NOCB_CPU) &&
+		     count != 0 && rcu_segcblist_empty(&rdp->cblist));
 
 	rcu_nocb_unlock_irqrestore(rdp, flags);
 
@@ -2564,8 +2569,9 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, bool lazy)
 		if (rcu_segcblist_empty(&rdp->cblist))
 			rcu_segcblist_init(&rdp->cblist);
 	}
-	rcu_nocb_lock(rdp);
-	was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
+	if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags))
+		return; // Enqueued onto ->nocb_bypass, so just leave.
+	/* If we get here, rcu_nocb_try_bypass() acquired ->nocb_lock. */
 	rcu_segcblist_enqueue(&rdp->cblist, head, lazy);
 	if (__is_kfree_rcu_offset((unsigned long)func))
 		trace_rcu_kfree_callback(rcu_state.name, head,
@@ -2839,6 +2845,7 @@ static void rcu_barrier_func(void *unused)
 	rdp->barrier_head.func = rcu_barrier_callback;
 	debug_rcu_head_queue(&rdp->barrier_head);
 	rcu_nocb_lock(rdp);
+	WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies));
 	if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head, 0)) {
 		atomic_inc(&rcu_state.barrier_cpu_count);
 	} else {
@@ -3192,6 +3199,7 @@ void rcutree_migrate_callbacks(int cpu)
 	my_rdp = this_cpu_ptr(&rcu_data);
 	my_rnp = my_rdp->mynode;
 	rcu_nocb_lock(my_rdp); /* irqs already disabled. */
+	WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies));
 	raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */
 	/* Leverage recent GPs and set GP for new callbacks. */
 	needwake = rcu_advance_cbs(my_rnp, rdp) ||
kernel/rcu/tree.h +18 -10
index 2c3e9068671c..e4df86db8137 100644
@@ -200,18 +200,26 @@ struct rcu_data {
 	atomic_t nocb_lock_contended;	/* Contention experienced. */
 	int nocb_defer_wakeup;		/* Defer wakeup of nocb_kthread. */
 	struct timer_list nocb_timer;	/* Enforce finite deferral. */
+	unsigned long nocb_gp_adv_time;	/* Last call_rcu() CB adv (jiffies). */
+
+	/* The following fields are used by call_rcu, hence own cacheline. */
+	raw_spinlock_t nocb_bypass_lock ____cacheline_internodealigned_in_smp;
+	struct rcu_cblist nocb_bypass;	/* Lock-contention-bypass CB list. */
+	unsigned long nocb_bypass_first; /* Time (jiffies) of first enqueue. */
+	unsigned long nocb_nobypass_last; /* Last ->cblist enqueue (jiffies). */
+	int nocb_nobypass_count;	/* # ->cblist enqueues at ^^^ time. */
 
 	/* The following fields are used by GP kthread, hence own cacheline. */
 	raw_spinlock_t nocb_gp_lock ____cacheline_internodealigned_in_smp;
-	bool nocb_gp_sleep;
-					/* Is the nocb GP thread asleep? */
+	struct timer_list nocb_bypass_timer; /* Force nocb_bypass flush. */
+	bool nocb_gp_sleep;		/* Is the nocb GP thread asleep? */
 	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */
 	bool nocb_cb_sleep;		/* Is the nocb CB thread asleep? */
 	struct task_struct *nocb_cb_kthread;
 	struct rcu_data *nocb_next_cb_rdp;
 					/* Next rcu_data in wakeup chain. */
 
-	/* The following fields are used by CB kthread, hence new cachline. */
+	/* The following fields are used by CB kthread, hence new cacheline. */
 	struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp;
 					/* GP rdp takes GP-end wakeups. */
 #endif /* #ifdef CONFIG_RCU_NOCB_CPU */
@@ -419,6 +427,10 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp);
 static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp);
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq);
 static void rcu_init_one_nocb(struct rcu_node *rnp);
+static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
+				  unsigned long j);
+static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
+				bool *was_alldone, unsigned long flags);
 static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty,
 				 unsigned long flags);
 static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp);
@@ -430,19 +442,15 @@ static void rcu_nocb_lock(struct rcu_data *rdp);
 static void rcu_nocb_unlock(struct rcu_data *rdp);
 static void rcu_nocb_unlock_irqrestore(struct rcu_data *rdp,
 				       unsigned long flags);
+static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp);
 #ifdef CONFIG_RCU_NOCB_CPU
 static void __init rcu_organize_nocb_kthreads(void);
 #define rcu_nocb_lock_irqsave(rdp, flags)				\
 do {									\
-	if (!rcu_segcblist_is_offloaded(&(rdp)->cblist)) {		\
+	if (!rcu_segcblist_is_offloaded(&(rdp)->cblist))		\
 		local_irq_save(flags);					\
-	} else if (!raw_spin_trylock_irqsave(&(rdp)->nocb_lock, (flags))) {\
-		atomic_inc(&(rdp)->nocb_lock_contended);		\
-		smp_mb__after_atomic(); /* atomic_inc() before lock. */	\
+	else								\
 		raw_spin_lock_irqsave(&(rdp)->nocb_lock, (flags));	\
-		smp_mb__before_atomic(); /* atomic_dec() after lock. */	\
-		atomic_dec(&(rdp)->nocb_lock_contended);		\
-	}								\
 } while (0)
 #else /* #ifdef CONFIG_RCU_NOCB_CPU */
 #define rcu_nocb_lock_irqsave(rdp, flags) local_irq_save(flags)
kernel/rcu/tree_plugin.h +329 -27

472 lines not on this page · raw message

Signature · 1 line
2.17.1
#19 lore raw

RFC PATCH 8/14 rcu/nocb: Reduce __call_rcu_nocb_wake() leaf rcu_node ->lock contention

In reply to Paul E. McKenney

Currently, __call_rcu_nocb_wake() advances callbacks each time that it detects excessive numbers of callbacks, though only if it succeeds in conditionally acquiring its leaf rcu_node structure's ->lock. Despite the conditional acquisition of ->lock, this does increase contention. This commit therefore avoids advancing callbacks unless there are callbacks in ->cblist whose grace period has completed and advancing has not yet been done during this jiffy.

Note that this decision does not take the presence of new callbacks into account. That is because on this code path, there will always be at least one new callback, namely the one we just enqueued.

---
 kernel/rcu/tree_plugin.h | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/rcu/tree_plugin.h +10 -3
index 02739366ef5d..efd7f6fa2790 100644
@@ -1869,6 +1869,8 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 				 unsigned long flags)
 				 __releases(rdp->nocb_lock)
 {
+	unsigned long cur_gp_seq;
+	unsigned long j;
 	long len;
 	struct task_struct *t;
 
@@ -1897,12 +1899,17 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,
 	} else if (len > rdp->qlen_last_fqs_check + qhimark) {
 		/* ... or if many callbacks queued. */
 		rdp->qlen_last_fqs_check = len;
-		if (rdp->nocb_cb_sleep ||
-		    !rcu_segcblist_ready_cbs(&rdp->cblist)) {
+		j = jiffies;
+		if (j != rdp->nocb_gp_adv_time &&
+		    rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
+		    rcu_seq_done(&rdp->mynode->gp_seq, cur_gp_seq)) {
 			rcu_advance_cbs_nowake(rdp->mynode, rdp);
+			rdp->nocb_gp_adv_time = j;
+		}
+		if (rdp->nocb_cb_sleep ||
+		    !rcu_segcblist_ready_cbs(&rdp->cblist))
 			wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE,
 					   TPS("WakeOvfIsDeferred"));
-		}
 		rcu_nocb_unlock_irqrestore(rdp, flags);
 	} else {
 		trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot"));
Signature · 1 line
2.17.1
maintainer
#26 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

maintainer
#27 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 11 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

maintainer
#2 lore raw

Re: [PATCH RFC tip/core/rcu 01/14] rcu/nocb: Atomic ->len field in rcu_segcblist structure

In reply to Paul E. McKenney
Quoted 9 lines · Paul E. McKenney
kernel/rcu/rcu_segcblist.c +9 -0
+/*
+ * Exchange the numeric length of the specified rcu_segcblist structure
+ * with the specified value.  This can cause the ->len field to disagree
+ * with the actual number of callbacks on the structure.  This exchange is
+ * fully ordered with respect to the callers accesses both before and after.
+ */
+long rcu_segcblist_xchg_len(struct rcu_segcblist *rsclp, long v)
+{
+#ifdef CONFIG_RCU_NOCB_CPU

kernel/rcu/rcu_segcblist.c struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp) · line 119

Quoted 10 lines
kernel/rcu/rcu_segcblist.c +10 -0
+	return atomic_long_xchg(&rsclp->len, v);
+#else
+	long ret = rsclp->len;
+
+	smp_mb(); /* Up to the caller! */
+	WRITE_ONCE(rsclp->len, v);
+	smp_mb(); /* Up to the caller! */
+	return ret;
+#endif
+}

That one's weird; for matching semantics the load needs to be between the memory barriers.

maintainer
#3 lore raw

Re: [PATCH RFC tip/core/rcu 01/14] rcu/nocb: Atomic ->len field in rcu_segcblist structure

Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Paul E. McKenney
kernel/rcu/rcu_segcblist.c +14 -0
+/*
+ * Exchange the numeric length of the specified rcu_segcblist structure
+ * with the specified value.  This can cause the ->len field to disagree
+ * with the actual number of callbacks on the structure.  This exchange is
+ * fully ordered with respect to the callers accesses both before and after.
+ */
+long rcu_segcblist_xchg_len(struct rcu_segcblist *rsclp, long v)
+{
+#ifdef CONFIG_RCU_NOCB_CPU
+	return atomic_long_xchg(&rsclp->len, v);
+#else
+	long ret = rsclp->len;
+
+	smp_mb(); /* Up to the caller! */
Quoted 8 lines

kernel/rcu/rcu_segcblist.c struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp) · line 119

Quoted 5 lines
kernel/rcu/rcu_segcblist.c +5 -0
+	WRITE_ONCE(rsclp->len, v);
+	smp_mb(); /* Up to the caller! */
+	return ret;
+#endif
+}

That one's weird; for matching semantics the load needs to be between the memory barriers.

Also, since you WRITE_ONCE() the thing, the load needs to be a READ_ONCE().

#28 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

Quoted 2 lines
Quoted 2 lines

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

Agreed. However, it is proving surprisingly hard to come up with a code sequence that has the effect of rcu_nocb without nohz_full. And rcu_nocb works just fine. With nohz_full also in place, I am decreasing the failure rate, but it still fails, perhaps a few times per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU system stubbornly refuses to fail. Good thing I kept the eight-CPU system around, I guess.)

When I arrive at some sequence of actions that actually work reliably, then by all means let's put it somewhere in the NO_HZ_FULL machinery!

Quoted 2 lines

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

There is no preemption because CONFIG_PREEMPT=n for the scenarios still having trouble. Yes, there are cond_resched() calls, but they don't do anything unless the appropriate flags are set, which won't always happen without the tick, apparently. Or without -something- that isn't always happening as it should.

Quoted 2 lines

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

I did try hitting all of the CPUs with resched_cpu(). Ten times on each CPU with a ten-jiffy wait between each. This might have decreased the probability of excessively long CPU-stopper waits by a factor of two or three, but it did not eliminate the excessively long waits.

What else should I try?

For example, are there any diagnostics I could collect, say from within the CPU stopper when things are taking too long? I see CPU-stopper delays in excess of five -minutes-, so this is anything but subtle.

Thanx, Paul

#5 lore raw

Re: [PATCH RFC tip/core/rcu 01/14] rcu/nocb: Atomic ->len field in rcu_segcblist structure

In reply to Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Paul E. McKenney
kernel/rcu/rcu_segcblist.c +14 -0
+/*
+ * Exchange the numeric length of the specified rcu_segcblist structure
+ * with the specified value.  This can cause the ->len field to disagree
+ * with the actual number of callbacks on the structure.  This exchange is
+ * fully ordered with respect to the callers accesses both before and after.
+ */
+long rcu_segcblist_xchg_len(struct rcu_segcblist *rsclp, long v)
+{
+#ifdef CONFIG_RCU_NOCB_CPU
+	return atomic_long_xchg(&rsclp->len, v);
+#else
+	long ret = rsclp->len;
+
+	smp_mb(); /* Up to the caller! */
Quoted 8 lines

kernel/rcu/rcu_segcblist.c struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp) · line 119

Quoted 5 lines
kernel/rcu/rcu_segcblist.c +5 -0
+	WRITE_ONCE(rsclp->len, v);
+	smp_mb(); /* Up to the caller! */
+	return ret;
+#endif
+}

That one's weird; for matching semantics the load needs to be between the memory barriers.

This is to match the concurrent version, which uses atomic_xchg().

Thanx, Paul

#4 lore raw

Re: [PATCH RFC tip/core/rcu 01/14] rcu/nocb: Atomic ->len field in rcu_segcblist structure

In reply to Peter Zijlstra
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Paul E. McKenney
kernel/rcu/rcu_segcblist.c +14 -0
+/*
+ * Exchange the numeric length of the specified rcu_segcblist structure
+ * with the specified value.  This can cause the ->len field to disagree
+ * with the actual number of callbacks on the structure.  This exchange is
+ * fully ordered with respect to the callers accesses both before and after.
+ */
+long rcu_segcblist_xchg_len(struct rcu_segcblist *rsclp, long v)
+{
+#ifdef CONFIG_RCU_NOCB_CPU
+	return atomic_long_xchg(&rsclp->len, v);
+#else
+	long ret = rsclp->len;
+
+	smp_mb(); /* Up to the caller! */
Quoted 8 lines

kernel/rcu/rcu_segcblist.c struct rcu_head *rcu_cblist_dequeue(struct rcu_cblist *rclp) · line 119

Quoted 5 lines
kernel/rcu/rcu_segcblist.c +5 -0
+	WRITE_ONCE(rsclp->len, v);
+	smp_mb(); /* Up to the caller! */
+	return ret;
+#endif
+}

That one's weird; for matching semantics the load needs to be between the memory barriers.

Quoted 2 lines

Also, since you WRITE_ONCE() the thing, the load needs to be a READ_ONCE().

Not in this case, because ->len is written only by the CPU in question in the !RCU_NOCB_CPU case.

It would not be hard to convince me that adding READ_ONCE() would be cheap and easy future-proofing, but Linus has objected to that sort of thing in the past.

Thanx, Paul

#29 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 42 lines · Paul E. McKenney
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

Quoted 2 lines
Quoted 2 lines

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

Agreed. However, it is proving surprisingly hard to come up with a code sequence that has the effect of rcu_nocb without nohz_full. And rcu_nocb works just fine. With nohz_full also in place, I am decreasing the failure rate, but it still fails, perhaps a few times per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU system stubbornly refuses to fail. Good thing I kept the eight-CPU system around, I guess.)

When I arrive at some sequence of actions that actually work reliably, then by all means let's put it somewhere in the NO_HZ_FULL machinery!

Quoted 2 lines

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

There is no preemption because CONFIG_PREEMPT=n for the scenarios still having trouble. Yes, there are cond_resched() calls, but they don't do anything unless the appropriate flags are set, which won't always happen without the tick, apparently. Or without -something- that isn't always happening as it should.

Quoted 2 lines

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

I did try hitting all of the CPUs with resched_cpu(). Ten times on each CPU with a ten-jiffy wait between each. This might have decreased the probability of excessively long CPU-stopper waits by a factor of two or three, but it did not eliminate the excessively long waits.

Quoted 5 lines

What else should I try?

For example, are there any diagnostics I could collect, say from within the CPU stopper when things are taking too long? I see CPU-stopper delays in excess of five -minutes-, so this is anything but subtle.

For whatever it is worth, the things on my list include using 25 rounds of resched_cpu() on each CPU with ten-jiffy wait between each (instead of merely 10 rounds), using waitqueues or some such to actually force a meaningful context switch on the other CPUs, etc.

Thanx, Paul

#30 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Paul E. McKenney
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

Quoted 2 lines
Quoted 2 lines

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

Agreed. However, it is proving surprisingly hard to come up with a code sequence that has the effect of rcu_nocb without nohz_full. And rcu_nocb works just fine. With nohz_full also in place, I am decreasing the failure rate, but it still fails, perhaps a few times per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU system stubbornly refuses to fail. Good thing I kept the eight-CPU system around, I guess.)

When I arrive at some sequence of actions that actually work reliably, then by all means let's put it somewhere in the NO_HZ_FULL machinery!

Quoted 2 lines

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

There is no preemption because CONFIG_PREEMPT=n for the scenarios still having trouble. Yes, there are cond_resched() calls, but they don't do anything unless the appropriate flags are set, which won't always happen without the tick, apparently. Or without -something- that isn't always happening as it should.

Quoted 2 lines

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

I did try hitting all of the CPUs with resched_cpu(). Ten times on each CPU with a ten-jiffy wait between each. This might have decreased the probability of excessively long CPU-stopper waits by a factor of two or three, but it did not eliminate the excessively long waits.

What else should I try?

Quoted 8 lines
Quoted 3 lines

For example, are there any diagnostics I could collect, say from within the CPU stopper when things are taking too long? I see CPU-stopper delays in excess of five -minutes-, so this is anything but subtle.

For whatever it is worth, the things on my list include using 25 rounds of resched_cpu() on each CPU with ten-jiffy wait between each (instead of merely 10 rounds), using waitqueues or some such to actually force a meaningful context switch on the other CPUs, etc.

Which appears to have reduced the bug rate by about a factor of two. (But statistics and all that.)

I am now trying the same test, but with CONFIG_PREEMPT=y and without quite so much hammering on the scheduler. This is keying off Peter's earlier mention of preemption. If this turns out to be solid, perhaps we outlaw CONFIG_PREEMPT=n && CONFIG_NO_HZ_FULL=y?

Thanx, Paul

maintainer
#33 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 22 lines · Paul E. McKenney
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

Quoted 2 lines
Quoted 2 lines

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

Agreed. However, it is proving surprisingly hard to come up with a code sequence that has the effect of rcu_nocb without nohz_full. And rcu_nocb works just fine. With nohz_full also in place, I am decreasing the failure rate, but it still fails, perhaps a few times per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU system stubbornly refuses to fail. Good thing I kept the eight-CPU system around, I guess.)

Quoted 2 lines

When I arrive at some sequence of actions that actually work reliably, then by all means let's put it somewhere in the NO_HZ_FULL machinery!

I'm confused; what are you arguing? The patch as proposed is just wrong, it needs to go.

Quoted 4 lines
Quoted 2 lines

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

There is no preemption because CONFIG_PREEMPT=n for the scenarios still

That doesn't make sense; even with CONFIG_PREEMPT=n we set TIF_NEED_RESCHED. We'll just not react to it as promptly (only explicit rescheduling points and return to userspace). Enabling the tick will not make any difference what so ever.

Tick based preemption will not 'fix' the lack of wakeup preemption. If the stop task wakeup didn't set TIF_NEED_RESCHED, the OTHER/CFS tick will not either.

Quoted 4 lines

having trouble. Yes, there are cond_resched() calls, but they don't do anything unless the appropriate flags are set, which won't always happen without the tick, apparently. Or without -something- that isn't always happening as it should.

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Quoted 8 lines
Quoted 2 lines

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

I did try hitting all of the CPUs with resched_cpu(). Ten times on each CPU with a ten-jiffy wait between each. This might have decreased the probability of excessively long CPU-stopper waits by a factor of two or three, but it did not eliminate the excessively long waits.

Quoted 5 lines

What else should I try?

For example, are there any diagnostics I could collect, say from within the CPU stopper when things are taking too long? I see CPU-stopper delays in excess of five -minutes-, so this is anything but subtle.

Catch the whole thing in a function trace?

The chain that should instantly set TIF_NEED_RESCHED:

stop_machine()

    stop_machine_cpuslocked()
      stop_cpus()
        __stop_cpus()
          queue_stop_cpus_work()
            cpu_stop_queue_work()
	      wake_up_q()
	        wake_up_process()

wake_up_process()

    try_to_wake_up()
      ttwu_queue()
        ttwu_queue_remote()
	  <- scheduler_ipi()
	    sched_ttwu_pending()
	      ttwu_do_activate()

        ttwu_do_activate()
	  activate_task()
	  ttwu_do_wakeup()
	    check_preempt_curr()
	      resched_curr()

You could frob some tracing into __stop_cpus(), before wait_for_completion(), at that point all the CPUs in @cpumask should either be running the stop task or have TIF_NEED_RESCHED set.

maintainer
#31 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 6 lines · Paul E. McKenney
Quoted 4 lines · Paul E. McKenney

For whatever it is worth, the things on my list include using 25 rounds of resched_cpu() on each CPU with ten-jiffy wait between each (instead of merely 10 rounds), using waitqueues or some such to actually force a meaningful context switch on the other CPUs, etc.

That really should not be needed. What are those other CPUs doing?

Quoted 2 lines

Which appears to have reduced the bug rate by about a factor of two. (But statistics and all that.)

Which is just weird..

Quoted 4 lines

I am now trying the same test, but with CONFIG_PREEMPT=y and without quite so much hammering on the scheduler. This is keying off Peter's earlier mention of preemption. If this turns out to be solid, perhaps we outlaw CONFIG_PREEMPT=n && CONFIG_NO_HZ_FULL=y?

CONFIG_PREEMPT=n should work just fine, _something_ is off.

#32 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Paul E. McKenney
Quoted 4 lines · Paul E. McKenney

For whatever it is worth, the things on my list include using 25 rounds of resched_cpu() on each CPU with ten-jiffy wait between each (instead of merely 10 rounds), using waitqueues or some such to actually force a meaningful context switch on the other CPUs, etc.

That really should not be needed. What are those other CPUs doing?

Excellent question. It would be really nice to have a CPU-stopper stall warning, wouldn't it? But who knows? Maybe I am the only one to have run into this. However, the comment in multi_cpu_stop() just before the call to touch_nmi_watchdog() leads me to believe otherwise. ;-)

Quoted 4 lines
Quoted 2 lines

Which appears to have reduced the bug rate by about a factor of two. (But statistics and all that.)

Which is just weird..

Indeed. Your point being?

Quoted 6 lines
Quoted 4 lines

I am now trying the same test, but with CONFIG_PREEMPT=y and without quite so much hammering on the scheduler. This is keying off Peter's earlier mention of preemption. If this turns out to be solid, perhaps we outlaw CONFIG_PREEMPT=n && CONFIG_NO_HZ_FULL=y?

CONFIG_PREEMPT=n should work just fine, _something_ is off.

Thank you, that is what I needed to know.

Thanx, Paul

#34 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Peter Zijlstra
Quoted 23 lines · Peter Zijlstra
Quoted 22 lines · Paul E. McKenney
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

Quoted 2 lines
Quoted 2 lines

This sounds wrong; should we be fixing sched_can_stop_tick() instead to return false when the stop task is runnable?

Agreed. However, it is proving surprisingly hard to come up with a code sequence that has the effect of rcu_nocb without nohz_full. And rcu_nocb works just fine. With nohz_full also in place, I am decreasing the failure rate, but it still fails, perhaps a few times per hour of TREE04 rcutorture on an eight-CPU system. (My 12-CPU system stubbornly refuses to fail. Good thing I kept the eight-CPU system around, I guess.)

Quoted 5 lines
Quoted 2 lines

When I arrive at some sequence of actions that actually work reliably, then by all means let's put it somewhere in the NO_HZ_FULL machinery!

I'm confused; what are you arguing? The patch as proposed is just wrong, it needs to go.

Eventually, sure. But one dragon at a time. Right now that dragon is "what is required to get multi_cpu_stop() to work in a timely fashioon". The "where does that code really go" dragon comes later.

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

And even without that; I don't understand how we're not instantly preempted the moment we enqueue the stop task.

There is no preemption because CONFIG_PREEMPT=n for the scenarios still

Quoted 8 lines

That doesn't make sense; even with CONFIG_PREEMPT=n we set TIF_NEED_RESCHED. We'll just not react to it as promptly (only explicit rescheduling points and return to userspace). Enabling the tick will not make any difference what so ever.

Tick based preemption will not 'fix' the lack of wakeup preemption. If the stop task wakeup didn't set TIF_NEED_RESCHED, the OTHER/CFS tick will not either.

Seems logical except for the fact that multi_cpu_stop() really is taking in excess of five minutes on a regular basis.

Quoted 7 lines
Quoted 4 lines

having trouble. Yes, there are cond_resched() calls, but they don't do anything unless the appropriate flags are set, which won't always happen without the tick, apparently. Or without -something- that isn't always happening as it should.

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 42 lines
Quoted 8 lines
Quoted 2 lines

Any enqueue, should go through check_preempt_curr() which will be an instant resched_curr() when we just woke the stop class.

I did try hitting all of the CPUs with resched_cpu(). Ten times on each CPU with a ten-jiffy wait between each. This might have decreased the probability of excessively long CPU-stopper waits by a factor of two or three, but it did not eliminate the excessively long waits.

Quoted 5 lines

What else should I try?

For example, are there any diagnostics I could collect, say from within the CPU stopper when things are taking too long? I see CPU-stopper delays in excess of five -minutes-, so this is anything but subtle.

Catch the whole thing in a function trace?

The chain that should instantly set TIF_NEED_RESCHED:

stop_machine()

    stop_machine_cpuslocked()
      stop_cpus()
        __stop_cpus()
          queue_stop_cpus_work()
            cpu_stop_queue_work()
	      wake_up_q()
	        wake_up_process()

wake_up_process()

    try_to_wake_up()
      ttwu_queue()
        ttwu_queue_remote()
	  <- scheduler_ipi()
	    sched_ttwu_pending()
	      ttwu_do_activate()

        ttwu_do_activate()
	  activate_task()
	  ttwu_do_wakeup()
	    check_preempt_curr()
	      resched_curr()
Quoted 3 lines

You could frob some tracing into __stop_cpus(), before wait_for_completion(), at that point all the CPUs in @cpumask should either be running the stop task or have TIF_NEED_RESCHED set.

Thank you, this should be quite helpful.

Thanx, Paul

maintainer
#35 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

#36 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Quoted 4 lines
Quoted 1 line

Understood???

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

Thanx, Paul

#37 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 7 lines
Quoted 2 lines

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

And I did eventually get a good trace. If I am interpreting this trace correctly, the torture_-135 task didn't get around to attempting to wake up all of the CPUs. I will try again, but this time with the sched_switch trace event enabled.

As a side note, enabling ftrace from the command line seems to interact badly with turning tracing off and on in the kernel, so I eventually resorted to trace_printk() in the functions of interest. The trace output is below, followed by the current diagnostic patch. Please note that I am -not- using the desperation hammer-the-scheduler patches.

More as I learn more!

Thanx, Paul

------------------------------------------------------------------------

[  280.918596] torture_-135     0.... 163481679us : __stop_cpus: __stop_cpus entered
[  280.918596] torture_-135     0.... 163481680us : queue_stop_cpus_work: queue_stop_cpus_work entered
[  280.918596] torture_-135     0.... 163481681us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.... 163481681us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.... 163481682us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.... 163481682us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0d... 163481682us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0d... 163481682us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] torture_-135     0d... 163481682us : activate_task: activate_task entered
[  280.918596] torture_-135     0d... 163481683us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] torture_-135     0d... 163481683us : check_preempt_curr: check_preempt_curr entered
[  280.918596] torture_-135     0d... 163481683us : resched_curr: resched_curr entered
[  280.918596] torture_-135     0.N.. 163481684us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.N.. 163481684us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481684us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481685us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481685us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0dN.. 163481685us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] torture_-135     0.N.. 163481695us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596]   <idle>-0       1d... 163481700us : scheduler_ipi: scheduler_ipi entered
[  280.918596]   <idle>-0       1d.h. 163481702us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596]   <idle>-0       1d.h. 163481702us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596]   <idle>-0       1d.h. 163481702us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596]   <idle>-0       1d.h. 163481703us : activate_task: activate_task entered
[  280.918596]   <idle>-0       1d.h. 163481703us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596]   <idle>-0       1d.h. 163481703us : check_preempt_curr: check_preempt_curr entered
[  280.918596]   <idle>-0       1d.h. 163481707us : resched_curr: resched_curr entered
[  280.918596] torture_-135     0.N.. 163481713us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481713us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481713us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481713us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0dN.. 163481714us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] torture_-135     0.N.. 163481722us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.N.. 163481723us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481723us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481723us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481723us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0dN.. 163481724us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596]   <idle>-0       1.N.. 163481730us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] torture_-135     0.N.. 163481732us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.N.. 163481732us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481732us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481733us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481733us : try_to_wake_up: ttwu_queue entered
[  280.918596] rcu_tort-130     3d... 163481733us : scheduler_ipi: scheduler_ipi entered
[  280.918596] torture_-135     0dN.. 163481733us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] rcu_tort-130     3d.h. 163481734us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] rcu_tort-130     3d.h. 163481734us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] rcu_tort-130     3d.h. 163481735us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] rcu_tort-130     3d.h. 163481735us : activate_task: activate_task entered
[  280.918596] rcu_tort-130     3d.h. 163481735us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] rcu_tort-130     3d.h. 163481735us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-130     3d.h. 163481735us : resched_curr: resched_curr entered
[  280.918596] torture_-135     0.N.. 163481742us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.N.. 163481742us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481742us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481742us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481743us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0dN.. 163481743us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] rcu_tort-128     4d... 163481743us : scheduler_ipi: scheduler_ipi entered
[  280.918596] rcu_tort-124     2d... 163481746us : scheduler_ipi: scheduler_ipi entered
[  280.918596] rcu_tort-128     4d.h. 163481746us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] rcu_tort-128     4d.h. 163481746us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] rcu_tort-124     2d.h. 163481747us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] rcu_tort-128     4d.h. 163481747us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] rcu_tort-124     2d.h. 163481747us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] rcu_tort-124     2d.h. 163481747us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] rcu_tort-128     4d.h. 163481747us : activate_task: activate_task entered
[  280.918596] rcu_tort-124     2d.h. 163481748us : activate_task: activate_task entered
[  280.918596] rcu_tort-128     4d.h. 163481748us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] rcu_tort-124     2d.h. 163481748us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] rcu_tort-128     4d.h. 163481748us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-124     2d.h. 163481748us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-128     4d.h. 163481748us : resched_curr: resched_curr entered
[  280.918596] rcu_tort-124     2d.h. 163481749us : resched_curr: resched_curr entered
[  280.918596] torture_-135     0.N.. 163481752us : cpu_stop_queue_work: cpu_stop_queue_work entered
[  280.918596] torture_-135     0.N.. 163481753us : wake_up_q: wake_up_q entered
[  280.918596] torture_-135     0.N.. 163481753us : wake_up_process: wake_up_process entered
[  280.918596] torture_-135     0.N.. 163481753us : try_to_wake_up: try_to_wake_up entered
[  280.918596] torture_-135     0dN.. 163481753us : try_to_wake_up: ttwu_queue entered
[  280.918596] torture_-135     0dN.. 163481753us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596]   <idle>-0       5d... 163481772us : scheduler_ipi: scheduler_ipi entered
[  280.918596]   <idle>-0       5d.h. 163481773us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596]   <idle>-0       5d.h. 163481774us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596]   <idle>-0       5d.h. 163481774us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596]   <idle>-0       5d.h. 163481775us : activate_task: activate_task entered
[  280.918596]   <idle>-0       5d.h. 163481776us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596]   <idle>-0       5d.h. 163481776us : check_preempt_curr: check_preempt_curr entered
[  280.918596]   <idle>-0       5d.h. 163481776us : resched_curr: resched_curr entered
[  280.918596]   <idle>-0       5.N.. 163481808us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-11      0..s1 163481828us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 163481829us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 163481829us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 163481829us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] migratio-14      1d..1 163481869us : scheduler_ipi: scheduler_ipi entered
[  280.918596] migratio-14      1d.h1 163481870us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-14      1d.h1 163481870us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] migratio-14      1d.h1 163481871us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-14      1d.h1 163481890us : activate_task: activate_task entered
[  280.918596] migratio-14      1d.h1 163481891us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-14      1d.h1 163481892us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0..s1 163483828us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 163483828us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 163483828us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 163483829us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 163483829us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 163483829us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 163483829us : check_preempt_curr: check_preempt_curr entered
[  280.918596]   <idle>-0       7d... 163483829us : scheduler_ipi: scheduler_ipi entered
[  280.918596]   <idle>-0       7d.h. 163486880us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596]   <idle>-0       7d.h. 163486881us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596]   <idle>-0       7d.h. 163486885us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596]   <idle>-0       7d.h. 163486885us : activate_task: activate_task entered
[  280.918596]   <idle>-0       7d.h. 163486885us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596]   <idle>-0       7d.h. 163486886us : check_preempt_curr: check_preempt_curr entered
[  280.918596]   <idle>-0       7d.h. 163486889us : resched_curr: resched_curr entered
[  280.918596]   <idle>-0       7.N.. 163486937us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-40      5d..1 278818221us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-11      0d.h1 278818380us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.h1 278818382us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.h1 278818384us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.h1 278818384us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] rcu_tort-123     3d... 278818385us : wake_up_process: wake_up_process entered
[  280.918596] rcu_tort-123     3d... 278818386us : try_to_wake_up: try_to_wake_up entered
[  280.918596] rcu_tort-123     3d... 278818387us : try_to_wake_up: ttwu_queue entered
[  280.918596] rcu_tort-123     3d... 278818387us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] migratio-33      4d.s1 278818403us : activate_task: activate_task entered
[  280.918596] migratio-33      4d.s1 278818405us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818416us : scheduler_ipi: scheduler_ipi entered
[  280.918596] migratio-11      0d.H1 278818417us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-11      0d.H1 278818417us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] migratio-11      0d.H1 278818418us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.H1 278818421us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.H1 278818423us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.H1 278818423us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0..s1 278818425us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 278818426us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818427us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818427us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818427us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818428us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818428us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-124     2d... 278818429us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818430us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818431us : try_to_wake_up: try_to_wake_up entered
[  280.918596] rcu_tort-124     2d... 278818431us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818431us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818432us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818432us : activate_task: activate_task entered
[  280.918596] rcu_tort-124     2d... 278818432us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818432us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818433us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-124     2d... 278818434us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818434us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818434us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818435us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818435us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818436us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818436us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818438us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818438us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818439us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818440us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0..s1 278818441us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 278818441us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818441us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818442us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818442us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818443us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818443us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0..s1 278818444us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 278818444us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818445us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818445us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818446us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818446us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818447us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0..s1 278818449us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 278818449us : try_to_wake_up: try_to_wake_up entered
[  280.918596] rcu_tort-126     2d... 278818451us : activate_task: activate_task entered
[  280.918596] rcu_tort-126     2d... 278818452us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-126     2d... 278818452us : resched_curr: resched_curr entered
[  280.918596] rcu_tort-126     2dN.. 278818453us : activate_task: activate_task entered
[  280.918596] rcu_tort-126     2dN.. 278818454us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-126     2dN.. 278818454us : activate_task: activate_task entered
[  280.918596] rcu_tort-126     2dN.. 278818455us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818466us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818466us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818467us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818467us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818468us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818469us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818469us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0..s1 278818481us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0..s1 278818481us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818482us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-11      0d.s1 278818483us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-11      0d.s1 278818483us : activate_task: activate_task entered
[  280.918596] migratio-11      0d.s1 278818484us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-11      0d.s1 278818484us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-11      0d.s1 278818485us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818485us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818487us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818487us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818488us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818488us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818489us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818490us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818491us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818491us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818492us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818492us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818493us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818493us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-11      0d.s1 278818494us : wake_up_process: wake_up_process entered
[  280.918596] migratio-11      0d.s1 278818495us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-62      0d... 278818512us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-62      0d... 278818512us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-62      0d... 278818513us : try_to_wake_up: ttwu_queue entered
[  280.918596] kworker/-62      0d... 278818514us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] kworker/-62      0d... 278818514us : activate_task: activate_task entered
[  280.918596] kworker/-62      0d... 278818515us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] kworker/-62      0d... 278818516us : check_preempt_curr: check_preempt_curr entered
[  280.918596] kworker/-161     0d... 278818518us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-161     0d... 278818519us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-161     0d... 278818519us : try_to_wake_up: ttwu_queue entered
[  280.918596] kworker/-161     0d... 278818519us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] kworker/-161     0d... 278818520us : activate_task: activate_task entered
[  280.918596] kworker/-161     0d... 278818520us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] kworker/-161     0d... 278818521us : check_preempt_curr: check_preempt_curr entered
[  280.918596] kworker/-161     0d... 278818522us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-161     0d... 278818522us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-161     0d... 278818523us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-161     0d... 278818523us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-161     0d... 278818524us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-161     0d... 278818524us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-161     0d... 278818526us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-161     0d... 278818526us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-177     0.... 278818628us : wake_up_q: wake_up_q entered
[  280.918596] kworker/-177     0d... 278818629us : wake_up_process: wake_up_process entered
[  280.918596] kworker/-177     0d... 278818629us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-177     0d... 278818630us : try_to_wake_up: ttwu_queue entered
[  280.918596] kworker/-177     0d... 278818630us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] kworker/-177     0d... 278818631us : activate_task: activate_task entered
[  280.918596] kworker/-177     0d... 278818632us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] kworker/-177     0d... 278818632us : check_preempt_curr: check_preempt_curr entered
[  280.918596] rcu_tort-131     2d.s. 278821050us : wake_up_process: wake_up_process entered
[  280.918596] rcu_tort-131     2d.s. 278821051us : try_to_wake_up: try_to_wake_up entered
[  280.918596] rcu_tort-131     2d.s. 278821052us : try_to_wake_up: ttwu_queue entered
[  280.918596] rcu_tort-131     2d.s. 278821052us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] rcu_tort-131     2d.s. 278821052us : activate_task: activate_task entered
[  280.918596] rcu_tort-131     2d.s. 278821053us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] rcu_tort-131     2d.s. 278821054us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-14      1dNs1 278821054us : scheduler_ipi: scheduler_ipi entered
[  280.918596] migratio-14      1dNH1 278821055us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] migratio-14      1dNH1 278821055us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] migratio-14      1dNH1 278821056us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] migratio-14      1dNH1 278821056us : activate_task: activate_task entered
[  280.918596] migratio-14      1dNH1 278821058us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] migratio-14      1dNH1 278821058us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-14      1dNs1 278821070us : activate_task: activate_task entered
[  280.918596] migratio-14      1dNs1 278821070us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-14      1dNs1 278821071us : activate_task: activate_task entered
[  280.918596] migratio-14      1dNs1 278821071us : check_preempt_curr: check_preempt_curr entered
[  280.918596] migratio-14      1dN.1 278821073us : try_to_wake_up: try_to_wake_up entered
[  280.918596] migratio-14      1dN.1 278821074us : try_to_wake_up: ttwu_queue entered
[  280.918596] migratio-14      1dN.1 278821075us : try_to_wake_up: ttwu_queue_remote entered
[  280.918596] kworker/-178     0d... 278821153us : scheduler_ipi: scheduler_ipi entered
[  280.918596] kworker/-178     0d.h. 278821154us : sched_ttwu_pending: sched_ttwu_pending entered
[  280.918596] kworker/-178     0d.h. 278821154us : sched_ttwu_pending: sched_ttwu_pending non-NULL llist
[  280.918596] kworker/-178     0d.h. 278821155us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] kworker/-178     0d.h. 278821155us : activate_task: activate_task entered
[  280.918596] kworker/-178     0d.h. 278821157us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] kworker/-178     0d.h. 278821157us : check_preempt_curr: check_preempt_curr entered
[  280.918596] kworker/-178     0d... 278821174us : try_to_wake_up: try_to_wake_up entered
[  280.918596] kworker/-178     0d... 278821175us : try_to_wake_up: ttwu_queue entered
[  280.918596] kworker/-178     0d... 278821175us : ttwu_do_activate.isra.108: ttwu_do_activate entered
[  280.918596] kworker/-178     0d... 278821176us : activate_task: activate_task entered
[  280.918596] kworker/-178     0d... 278821176us : ttwu_do_wakeup.isra.107: ttwu_do_wakeup entered
[  280.918596] kworker/-178     0d... 278821177us : check_preempt_curr: check_preempt_curr entered

------------------------------------------------------------------------

kernel/rcu/tree.c +1 -0
index ce00b442ced0..1a50ed258ef0 100644
@@ -3569,6 +3569,7 @@ void __init rcu_init(void)
 	rcu_par_gp_wq = alloc_workqueue("rcu_par_gp", WQ_MEM_RECLAIM, 0);
 	WARN_ON(!rcu_par_gp_wq);
 	srcu_init();
+	tracing_off();
 }
 
 #include "tree_stall.h"
kernel/sched/core.c +13 -0
index 0b22e55cebe8..6949ae27fae5 100644
@@ -479,6 +479,7 @@ void wake_up_q(struct wake_q_head *head)
 {
 	struct wake_q_node *node = head->first;
 
+	trace_printk("%s entered\n", __func__);
 	while (node != WAKE_Q_TAIL) {
 		struct task_struct *task;
 
@@ -509,6 +510,7 @@ void resched_curr(struct rq *rq)
 	struct task_struct *curr = rq->curr;
 	int cpu;
 
+	trace_printk("%s entered\n", __func__);
 	lockdep_assert_held(&rq->lock);
 
 	if (test_tsk_need_resched(curr))
@@ -1197,6 +1199,7 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 void activate_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	trace_printk("%s entered\n", __func__);
 	if (task_contributes_to_load(p))
 		rq->nr_uninterruptible--;
 
@@ -1298,6 +1301,7 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)
 {
 	const struct sched_class *class;
 
+	trace_printk("%s entered\n", __func__);
 	if (p->sched_class == rq->curr->sched_class) {
 		rq->curr->sched_class->check_preempt_curr(rq, p, flags);
 	} else {
@@ -2097,6 +2101,7 @@ ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
 static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
 			   struct rq_flags *rf)
 {
+	trace_printk("%s entered\n", __func__);
 	check_preempt_curr(rq, p, wake_flags);
 	p->state = TASK_RUNNING;
 	trace_sched_wakeup(p);
@@ -2132,6 +2137,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 {
 	int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
 
+	trace_printk("%s entered\n", __func__);
 	lockdep_assert_held(&rq->lock);
 
 #ifdef CONFIG_SMP
@@ -2178,9 +2184,11 @@ void sched_ttwu_pending(void)
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
+	trace_printk("%s entered\n", __func__);
 	if (!llist)
 		return;
 
+	trace_printk("%s non-NULL llist\n", __func__);
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
@@ -2192,6 +2200,7 @@ void sched_ttwu_pending(void)
 
 void scheduler_ipi(void)
 {
+	trace_printk("%s entered\n", __func__);
 	/*
 	 * Fold TIF_NEED_RESCHED into the preempt_count; anybody setting
 	 * TIF_NEED_RESCHED remotely (for the first time) will also send
@@ -2232,6 +2241,7 @@ static void ttwu_queue_remote(struct task_struct *p, int cpu, int wake_flags)
 {
 	struct rq *rq = cpu_rq(cpu);
 
+	trace_printk("%s entered\n", __func__);
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
@@ -2277,6 +2287,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
 	struct rq *rq = cpu_rq(cpu);
 	struct rq_flags rf;
 
+	trace_printk("%s entered\n", __func__);
 #if defined(CONFIG_SMP)
 	if (sched_feat(TTWU_QUEUE) && !cpus_share_cache(smp_processor_id(), cpu)) {
 		sched_clock_cpu(cpu); /* Sync clocks across CPUs */
@@ -2399,6 +2410,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 	unsigned long flags;
 	int cpu, success = 0;
 
+	trace_printk("%s entered\n", __func__);
 	preempt_disable();
 	if (p == current) {
 		/*
@@ -2545,6 +2557,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
  */
 int wake_up_process(struct task_struct *p)
 {
+	trace_printk("%s entered\n", __func__);
 	return try_to_wake_up(p, TASK_NORMAL, 0);
 }
 EXPORT_SYMBOL(wake_up_process);
kernel/stop_machine.c +10 -0
index 5c2b2f90fae1..d3441acabc80 100644
@@ -80,6 +80,7 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work)
 	unsigned long flags;
 	bool enabled;
 
+	trace_printk("%s entered\n", __func__);
 	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
@@ -382,6 +383,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,
 	 * preempted by a stopper which might wait for other stoppers
 	 * to enter @fn which can lead to deadlock.
 	 */
+	trace_printk("%s entered\n", __func__);
 	preempt_disable();
 	stop_cpus_in_progress = true;
 	for_each_cpu(cpu, cpumask) {
@@ -402,11 +404,17 @@ static int __stop_cpus(const struct cpumask *cpumask,
 		       cpu_stop_fn_t fn, void *arg)
 {
 	struct cpu_stop_done done;
+	unsigned long j = jiffies;
 
+	tracing_on();
+	trace_printk("%s entered\n", __func__);
 	cpu_stop_init_done(&done, cpumask_weight(cpumask));
 	if (!queue_stop_cpus_work(cpumask, fn, arg, &done))
 		return -ENOENT;
 	wait_for_completion(&done.completion);
+	tracing_off();
+	if (time_after(jiffies, j + HZ * 20))
+		ftrace_dump(DUMP_ALL);
 	return done.ret;
 }
 
@@ -442,6 +450,7 @@ int stop_cpus(const struct cpumask *cpumask, cpu_stop_fn_t fn, void *arg)
 {
 	int ret;
 
+	trace_printk("%s entered\n", __func__);
 	/* static works are used, process one request at a time */
 	mutex_lock(&stop_cpus_mutex);
 	ret = __stop_cpus(cpumask, fn, arg);
@@ -599,6 +608,7 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 		.active_cpus = cpus,
 	};
 
+	trace_printk("%s: entered\n", __func__);
 	lockdep_assert_cpus_held();
 
 	if (!stop_machine_initialized) {
#7 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

In reply to Paul E. McKenney
Quoted 9 lines · Paul E. McKenney

Use of the rcu_data structure's segmented ->cblist for no-CBs CPUs takes advantage of unrelated grace periods, thus reducing the memory footprint in the face of floods of call_rcu() invocations. However, the ->cblist field is a more-complex rcu_segcblist structure which must be protected via locking. Even though there are only three entities which can acquire this lock (the CPU invoking call_rcu(), the no-CBs grace-period kthread, and the no-CBs callbacks kthread), the contention on this lock is excessive under heavy stress.

Quoted 7 lines

This commit therefore greatly reduces contention by provisioning an rcu_cblist structure field named ->nocb_bypass within the rcu_data structure. Each no-CBs CPU is permitted only a limited number of enqueues onto the ->cblist per jiffy, controlled by a new nocb_nobypass_lim_per_jiffy kernel boot parameter that defaults to about 16 enqueues per millisecond (16 * 1000 / HZ). When that limit is exceeded, the CPU instead enqueues onto the new ->nocb_bypass.

Looks quite interesting. I am guessing the not-no-CB (regular) enqueues don't need to use the same technique because both enqueues / callback execution are happening on same CPU..

Still looking through patch but I understood the basic idea. Some nits below:

[snip]

Quoted 6 lines
kernel/rcu/tree.h +0 -0
index 2c3e9068671c..e4df86db8137 100644
@@ -200,18 +200,26 @@ struct rcu_data {
 	atomic_t nocb_lock_contended;	/* Contention experienced. */

kernel/rcu/tree.h struct rcu_data { · line 210

Quoted 10 lines
kernel/rcu/tree.h +8 -0
 	int nocb_defer_wakeup;		/* Defer wakeup of nocb_kthread. */
 	struct timer_list nocb_timer;	/* Enforce finite deferral. */
+	unsigned long nocb_gp_adv_time;	/* Last call_rcu() CB adv (jiffies). */
+
+	/* The following fields are used by call_rcu, hence own cacheline. */
+	raw_spinlock_t nocb_bypass_lock ____cacheline_internodealigned_in_smp;
+	struct rcu_cblist nocb_bypass;	/* Lock-contention-bypass CB list. */
+	unsigned long nocb_bypass_first; /* Time (jiffies) of first enqueue. */
+	unsigned long nocb_nobypass_last; /* Last ->cblist enqueue (jiffies). */
+	int nocb_nobypass_count;	/* # ->cblist enqueues at ^^^ time. */

Can these and below fields be ifdef'd out if !CONFIG_RCU_NOCB_CPU so as to keep the size of struct smaller for benefit of systems that don't use NOCB?

kernel/rcu/tree.h struct rcu_data { · line 215

Quoted 7 lines
kernel/rcu/tree.h +2 -2
 	/* The following fields are used by GP kthread, hence own cacheline. */
 	raw_spinlock_t nocb_gp_lock ____cacheline_internodealigned_in_smp;
-	bool nocb_gp_sleep;
-					/* Is the nocb GP thread asleep? */
+	struct timer_list nocb_bypass_timer; /* Force nocb_bypass flush. */
+	bool nocb_gp_sleep;		/* Is the nocb GP thread asleep? */

And these too, I think.

Quoted 11 lines
kernel/rcu/tree.h +1 -1
 	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */
 	bool nocb_cb_sleep;		/* Is the nocb CB thread asleep? */
 	struct task_struct *nocb_cb_kthread;
 	struct rcu_data *nocb_next_cb_rdp;
 					/* Next rcu_data in wakeup chain. */
 
-	/* The following fields are used by CB kthread, hence new cachline. */
+	/* The following fields are used by CB kthread, hence new cacheline. */
 	struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp;
 					/* GP rdp takes GP-end wakeups. */
 #endif /* #ifdef CONFIG_RCU_NOCB_CPU */

[snip]

Quoted 32 lines
kernel/rcu/tree.h +32 -0
+static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j)
+{
+	rcu_lockdep_assert_cblist_protected(rdp);
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist) ||
+	    !rcu_nocb_bypass_trylock(rdp))
+		return;
+	WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j));
+}
+
+/*
+ * See whether it is appropriate to use the ->nocb_bypass list in order
+ * to control contention on ->nocb_lock.  A limited number of direct
+ * enqueues are permitted into ->cblist per jiffy.  If ->nocb_bypass
+ * is non-empty, further callbacks must be placed into ->nocb_bypass,
+ * otherwise rcu_barrier() breaks.  Use rcu_nocb_flush_bypass() to switch
+ * back to direct use of ->cblist.  However, ->nocb_bypass should not be
+ * used if ->cblist is empty, because otherwise callbacks can be stranded
+ * on ->nocb_bypass because we cannot count on the current CPU ever again
+ * invoking call_rcu().  The general rule is that if ->nocb_bypass is
+ * non-empty, the corresponding no-CBs grace-period kthread must not be
+ * in an indefinite sleep state.
+ *
+ * Finally, it is not permitted to use the bypass during early boot,
+ * as doing so would confuse the auto-initialization code.  Besides
+ * which, there is no point in worrying about lock contention while
+ * there is only one CPU in operation.
+ */
+static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
+				bool *was_alldone, unsigned long flags)
+{
+	unsigned long c;
+	unsigned long cur_gp_seq;

kernel/rcu/tree.h

Quoted 10 lines
kernel/rcu/tree.h +10 -0
+	unsigned long j = jiffies;
+	long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
+
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
+		*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
+		return false; /* Not offloaded, no bypassing. */
+	}
+	lockdep_assert_irqs_disabled();
+
+	// Don't use ->nocb_bypass during early boot.

Very minor nit: comment style should be /* */

thanks,

- Joel

[snip]

#8 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

Quoted 47 lines · Joel Fernandes
Quoted 9 lines · Paul E. McKenney

Use of the rcu_data structure's segmented ->cblist for no-CBs CPUs takes advantage of unrelated grace periods, thus reducing the memory footprint in the face of floods of call_rcu() invocations. However, the ->cblist field is a more-complex rcu_segcblist structure which must be protected via locking. Even though there are only three entities which can acquire this lock (the CPU invoking call_rcu(), the no-CBs grace-period kthread, and the no-CBs callbacks kthread), the contention on this lock is excessive under heavy stress.

Quoted 7 lines

This commit therefore greatly reduces contention by provisioning an rcu_cblist structure field named ->nocb_bypass within the rcu_data structure. Each no-CBs CPU is permitted only a limited number of enqueues onto the ->cblist per jiffy, controlled by a new nocb_nobypass_lim_per_jiffy kernel boot parameter that defaults to about 16 enqueues per millisecond (16 * 1000 / HZ). When that limit is exceeded, the CPU instead enqueues onto the new ->nocb_bypass.

Looks quite interesting. I am guessing the not-no-CB (regular) enqueues don't need to use the same technique because both enqueues / callback execution are happening on same CPU..

Still looking through patch but I understood the basic idea. Some nits below:

[snip]

Quoted 6 lines
kernel/rcu/tree.h +0 -0
index 2c3e9068671c..e4df86db8137 100644
@@ -200,18 +200,26 @@ struct rcu_data {
 	atomic_t nocb_lock_contended;	/* Contention experienced. */

kernel/rcu/tree.h struct rcu_data { · line 210

Quoted 10 lines
kernel/rcu/tree.h +8 -0
 	int nocb_defer_wakeup;		/* Defer wakeup of nocb_kthread. */
 	struct timer_list nocb_timer;	/* Enforce finite deferral. */
+	unsigned long nocb_gp_adv_time;	/* Last call_rcu() CB adv (jiffies). */
+
+	/* The following fields are used by call_rcu, hence own cacheline. */
+	raw_spinlock_t nocb_bypass_lock ____cacheline_internodealigned_in_smp;
+	struct rcu_cblist nocb_bypass;	/* Lock-contention-bypass CB list. */
+	unsigned long nocb_bypass_first; /* Time (jiffies) of first enqueue. */
+	unsigned long nocb_nobypass_last; /* Last ->cblist enqueue (jiffies). */
+	int nocb_nobypass_count;	/* # ->cblist enqueues at ^^^ time. */

Can these and below fields be ifdef'd out if !CONFIG_RCU_NOCB_CPU so as to keep the size of struct smaller for benefit of systems that don't use NOCB?

Quoted 1 line
Quoted 8 lines

kernel/rcu/tree.h struct rcu_data { · line 215

Quoted 6 lines
kernel/rcu/tree.h +2 -2
 	/* The following fields are used by GP kthread, hence own cacheline. */
 	raw_spinlock_t nocb_gp_lock ____cacheline_internodealigned_in_smp;
-	bool nocb_gp_sleep;
-					/* Is the nocb GP thread asleep? */
+	struct timer_list nocb_bypass_timer; /* Force nocb_bypass flush. */
+	bool nocb_gp_sleep;		/* Is the nocb GP thread asleep? */

And these too, I think.

Please ignore this comment, I missed that these were already ifdef'd out since it did not appear in the diff.

thanks!

#9 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

In reply to Joel Fernandes
Quoted 18 lines · Joel Fernandes
Quoted 9 lines · Paul E. McKenney

Use of the rcu_data structure's segmented ->cblist for no-CBs CPUs takes advantage of unrelated grace periods, thus reducing the memory footprint in the face of floods of call_rcu() invocations. However, the ->cblist field is a more-complex rcu_segcblist structure which must be protected via locking. Even though there are only three entities which can acquire this lock (the CPU invoking call_rcu(), the no-CBs grace-period kthread, and the no-CBs callbacks kthread), the contention on this lock is excessive under heavy stress.

Quoted 7 lines

This commit therefore greatly reduces contention by provisioning an rcu_cblist structure field named ->nocb_bypass within the rcu_data structure. Each no-CBs CPU is permitted only a limited number of enqueues onto the ->cblist per jiffy, controlled by a new nocb_nobypass_lim_per_jiffy kernel boot parameter that defaults to about 16 enqueues per millisecond (16 * 1000 / HZ). When that limit is exceeded, the CPU instead enqueues onto the new ->nocb_bypass.

Quoted 3 lines

Looks quite interesting. I am guessing the not-no-CB (regular) enqueues don't need to use the same technique because both enqueues / callback execution are happening on same CPU..

That is the theory! ;-)

Quoted 14 lines

Still looking through patch but I understood the basic idea. Some nits below:

[snip]

Quoted 11 lines
kernel/rcu/tree.h +3 -0
index 2c3e9068671c..e4df86db8137 100644
@@ -200,18 +200,26 @@ struct rcu_data {
 	atomic_t nocb_lock_contended;	/* Contention experienced. */
 	int nocb_defer_wakeup;		/* Defer wakeup of nocb_kthread. */
 	struct timer_list nocb_timer;	/* Enforce finite deferral. */
+	unsigned long nocb_gp_adv_time;	/* Last call_rcu() CB adv (jiffies). */
+
+	/* The following fields are used by call_rcu, hence own cacheline. */
Quoted 8 lines

kernel/rcu/tree.h struct rcu_data { · line 210

Quoted 5 lines
kernel/rcu/tree.h +5 -0
+	raw_spinlock_t nocb_bypass_lock ____cacheline_internodealigned_in_smp;
+	struct rcu_cblist nocb_bypass;	/* Lock-contention-bypass CB list. */
+	unsigned long nocb_bypass_first; /* Time (jiffies) of first enqueue. */
+	unsigned long nocb_nobypass_last; /* Last ->cblist enqueue (jiffies). */
+	int nocb_nobypass_count;	/* # ->cblist enqueues at ^^^ time. */

Can these and below fields be ifdef'd out if !CONFIG_RCU_NOCB_CPU so as to keep the size of struct smaller for benefit of systems that don't use NOCB?

Please see below...

Quoted 16 lines

kernel/rcu/tree.h struct rcu_data { · line 215

Quoted 6 lines
kernel/rcu/tree.h +2 -2
 	/* The following fields are used by GP kthread, hence own cacheline. */
 	raw_spinlock_t nocb_gp_lock ____cacheline_internodealigned_in_smp;
-	bool nocb_gp_sleep;
-					/* Is the nocb GP thread asleep? */
+	struct timer_list nocb_bypass_timer; /* Force nocb_bypass flush. */
+	bool nocb_gp_sleep;		/* Is the nocb GP thread asleep? */

And these too, I think.

Quoted 6 lines
kernel/rcu/tree.h +0 -0
 	struct swait_queue_head nocb_gp_wq; /* For nocb kthreads to sleep on. */
 	bool nocb_cb_sleep;		/* Is the nocb CB thread asleep? */
 	struct task_struct *nocb_cb_kthread;
 	struct rcu_data *nocb_next_cb_rdp;
 					/* Next rcu_data in wakeup chain. */
 
Quoted 5 lines

kernel/rcu/tree.h struct rcu_data { · line 225

Quoted 5 lines
kernel/rcu/tree.h +1 -1
-	/* The following fields are used by CB kthread, hence new cachline. */
+	/* The following fields are used by CB kthread, hence new cacheline. */
 	struct rcu_data *nocb_gp_rdp ____cacheline_internodealigned_in_smp;
 					/* GP rdp takes GP-end wakeups. */
 #endif /* #ifdef CONFIG_RCU_NOCB_CPU */

I believe that they in fact are all under CONFIG_RCU_NOCB_CPU.

Quoted 37 lines

[snip]

Quoted 36 lines
kernel/rcu/tree.h +36 -0
+static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j)
+{
+	rcu_lockdep_assert_cblist_protected(rdp);
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist) ||
+	    !rcu_nocb_bypass_trylock(rdp))
+		return;
+	WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j));
+}
+
+/*
+ * See whether it is appropriate to use the ->nocb_bypass list in order
+ * to control contention on ->nocb_lock.  A limited number of direct
+ * enqueues are permitted into ->cblist per jiffy.  If ->nocb_bypass
+ * is non-empty, further callbacks must be placed into ->nocb_bypass,
+ * otherwise rcu_barrier() breaks.  Use rcu_nocb_flush_bypass() to switch
+ * back to direct use of ->cblist.  However, ->nocb_bypass should not be
+ * used if ->cblist is empty, because otherwise callbacks can be stranded
+ * on ->nocb_bypass because we cannot count on the current CPU ever again
+ * invoking call_rcu().  The general rule is that if ->nocb_bypass is
+ * non-empty, the corresponding no-CBs grace-period kthread must not be
+ * in an indefinite sleep state.
+ *
+ * Finally, it is not permitted to use the bypass during early boot,
+ * as doing so would confuse the auto-initialization code.  Besides
+ * which, there is no point in worrying about lock contention while
+ * there is only one CPU in operation.
+ */
+static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
+				bool *was_alldone, unsigned long flags)
+{
+	unsigned long c;
+	unsigned long cur_gp_seq;
+	unsigned long j = jiffies;
+	long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass);
+
+	if (!rcu_segcblist_is_offloaded(&rdp->cblist)) {
Quoted 8 lines

kernel/rcu/tree.h

Quoted 6 lines
kernel/rcu/tree.h +6 -0
+		*was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist);
+		return false; /* Not offloaded, no bypassing. */
+	}
+	lockdep_assert_irqs_disabled();
+
+	// Don't use ->nocb_bypass during early boot.

Very minor nit: comment style should be /* */

I thought that Linus said that "//" was now OK. Am I confused?

Thanx, Paul

#10 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

Quoted 5 lines · On Tue, 6 Aug 2019 17:35:01 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 3 lines

kernel/rcu/tree_plugin.h static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, · line 1771

Quoted 1 line
kernel/rcu/tree_plugin.h +1 -0
+	// Don't use ->nocb_bypass during early boot.  

Very minor nit: comment style should be /* */

I thought that Linus said that "//" was now OK. Am I confused?

Have a link?

-- Steve

#11 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Tue, 6 Aug 2019 17:35:01 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 3 lines

kernel/rcu/tree_plugin.h static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, · line 1771

Quoted 1 line
kernel/rcu/tree_plugin.h +1 -0
+	// Don't use ->nocb_bypass during early boot.  

Very minor nit: comment style should be /* */

I thought that Linus said that "//" was now OK. Am I confused?

Have a link?

https://lkml.org/lkml/2016/7/8/625

Thanx, Paul

#12 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

Quoted 8 lines · On Tue, 6 Aug 2019 18:17:07 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Tue, 6 Aug 2019 17:35:01 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 3 lines

kernel/rcu/tree_plugin.h static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, · line 1771

Quoted 1 line
kernel/rcu/tree_plugin.h +1 -0
+	// Don't use ->nocb_bypass during early boot.    

Very minor nit: comment style should be /* */

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

I thought that Linus said that "//" was now OK. Am I confused?

Have a link?

https://lkml.org/lkml/2016/7/8/625

The (c) form is particularly good for things like enum or structure member comments at the end of code, where you might want to align things up, but the ending comment marker ends up being visually pretty distracting (and lining _that_ up is too much make-believe work).

I think it's still for special occasions, and the above example doesn't look like one of them ;-)

I basically avoid the '//' comment, as it just adds inconstancy.

-- Steve

#13 lore raw

Re: [PATCH RFC tip/core/rcu 02/14] rcu/nocb: Add bypass callback queueing

Quoted 17 lines · Steven Rostedt
Quoted 8 lines · On Tue, 6 Aug 2019 18:17:07 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Tue, 6 Aug 2019 17:35:01 -0700 "Paul E. McKenney" <[email protected]>…
Quoted 3 lines

kernel/rcu/tree_plugin.h static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, · line 1771

Quoted 1 line
kernel/rcu/tree_plugin.h +1 -0
+	// Don't use ->nocb_bypass during early boot.    

Very minor nit: comment style should be /* */

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

I thought that Linus said that "//" was now OK. Am I confused?

Have a link?

https://lkml.org/lkml/2016/7/8/625

Quoted 7 lines

The (c) form is particularly good for things like enum or structure member comments at the end of code, where you might want to align things up, but the ending comment marker ends up being visually pretty distracting (and lining _that_ up is too much make-believe work).

I think it's still for special occasions, and the above example doesn't look like one of them ;-)

It does say "particularly good for", not "only good for. ;-)

Quoted 1 line

I basically avoid the '//' comment, as it just adds inconstancy.

It saves me two whacks on the shift key and three whacks on other keys. ;-)

Thanx, Paul

#38 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 27 lines · Paul E. McKenney
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 7 lines
Quoted 2 lines

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

And I did eventually get a good trace. If I am interpreting this trace correctly, the torture_-135 task didn't get around to attempting to wake up all of the CPUs. I will try again, but this time with the sched_switch trace event enabled.

Quoted 7 lines

As a side note, enabling ftrace from the command line seems to interact badly with turning tracing off and on in the kernel, so I eventually resorted to trace_printk() in the functions of interest. The trace output is below, followed by the current diagnostic patch. Please note that I am -not- using the desperation hammer-the-scheduler patches.

More as I learn more!

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Thanx, Paul

------------------------------------------------------------------------

This trace is taken after an RCU CPU stall warning following the start of CPU 4 going offline:

[ 2579.977765] Unregister pv shared memory for cpu 4

Here is the trace, trimming out the part from earlier CPU-hotplug operations:

[ 2813.040289] torture_-135     0.... 2578022804us : __stop_cpus: CPUs 0-7 online
	All eight CPUs are online.
[ 2813.040289] torture_-135     0.... 2578022805us : queue_stop_cpus_work: entered
[ 2813.040289] torture_-135     0.... 2578022805us : cpu_stop_queue_work: entered for CPU 0
[ 2813.040289] torture_-135     0.... 2578022806us : wake_up_q: entered
[ 2813.040289] torture_-135     0.... 2578022806us : wake_up_process: entered
[ 2813.040289] torture_-135     0.... 2578022806us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0d... 2578022806us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0d... 2578022807us : ttwu_do_activate.isra.108: entered
[ 2813.040289] torture_-135     0d... 2578022807us : activate_task: entered
[ 2813.040289] torture_-135     0d... 2578022807us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] torture_-135     0d... 2578022807us : check_preempt_curr: entered
[ 2813.040289] torture_-135     0d... 2578022807us : resched_curr: entered
[ 2813.040289] torture_-135     0.N.. 2578022808us : cpu_stop_queue_work: entered for CPU 1
[ 2813.040289] torture_-135     0.N.. 2578022809us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022809us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022809us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022809us : try_to_wake_up: entered
	Note: trace_printk() is confused by inlining.
[ 2813.040289] torture_-135     0dN.. 2578022810us : try_to_wake_up: entered, CPU 1
	So the above is really ttwu_queue_remote().
	We are running on CPU 0, so presumably don't need to IPI it.
	We have IPIed CPU 1.
[ 2813.040289] torture_-135     0.N.. 2578022819us : cpu_stop_queue_work: entered for CPU 2
[ 2813.040289] torture_-135     0.N.. 2578022820us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022820us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022820us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022820us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022821us : try_to_wake_up: entered, CPU 2
	We have IPIed CPUs 1-2.
[ 2813.040289] rcu_tort-129     1d... 2578022821us : scheduler_ipi: entered
	CPU 1 got its IPI, so CPUs 1-2 IPIed and CPU 1 received IPI.
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : sched_ttwu_pending: entered
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : ttwu_do_activate.isra.108: entered
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : activate_task: entered
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-129     1d.h. 2578022821us : resched_curr: entered
[ 2813.040289] rcu_tort-129     1d... 2578022821us : sched_switch: prev_comm=rcu_torture_rea prev_pid=129 prev_prio=139 prev_state=R+ ==> next_comm=migration/1 next_pid=14 next_prio=0
	CPU 1 has switched to its migration kthread.
[ 2813.040289] torture_-135     0.N.. 2578022830us : cpu_stop_queue_work: entered for CPU 3
[ 2813.040289] torture_-135     0.N.. 2578022831us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022831us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022831us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022831us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022832us : try_to_wake_up: entered, CPU 3
	We have IPIed CPUs 1-3.
[ 2813.040289] rcu_tort-126     2d... 2578022832us : scheduler_ipi: entered
	CPU 2 got its IPI, so CPUs 1-3 IPIed and CPUs 1-2 received IPI.
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : sched_ttwu_pending: entered
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : ttwu_do_activate.isra.108: entered
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : activate_task: entered
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-126     2d.h. 2578022832us : resched_curr: entered
[ 2813.040289] torture_-135     0.N.. 2578022841us : cpu_stop_queue_work: entered for CPU 4
[ 2813.040289] torture_-135     0.N.. 2578022841us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022841us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022841us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022842us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022842us : try_to_wake_up: entered, CPU 4
	We have IPIed CPUs 1-4.
[ 2813.040289] rcu_tort-128     3d... 2578022842us : scheduler_ipi: entered
	CPU 3 got its IPI, so CPUs 1-4 IPIed and CPUs 1-3 received IPI.
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : sched_ttwu_pending: entered
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : ttwu_do_activate.isra.108: entered
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : activate_task: entered
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-128     3d.h. 2578022842us : resched_curr: entered
[ 2813.040289] torture_-135     0.N.. 2578022853us : cpu_stop_queue_work: entered for CPU 5
[ 2813.040289] torture_-135     0.N.. 2578022853us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022853us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022854us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022854us : try_to_wake_up: entered
[ 2813.040289] torture_-135     0dN.. 2578022856us : try_to_wake_up: entered, CPU 5
	We have IPIed CPUs 1-5.
[ 2813.040289]   <idle>-0       4d... 2578022863us : scheduler_ipi: entered
	CPU 4 got its IPI, so CPUs 1-5 IPIed and CPUs 1-4 received IPI.
[ 2813.040289]   <idle>-0       4d.h. 2578022865us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       4d.h. 2578022865us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]   <idle>-0       4d.h. 2578022866us : ttwu_do_activate.isra.108: entered
[ 2813.040289]   <idle>-0       4d.h. 2578022866us : activate_task: entered
[ 2813.040289]   <idle>-0       4d.h. 2578022867us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-126     2d... 2578022867us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=migration/2 next_pid=21 next_prio=0
	CPUs 1-2 have switched to their migration kthreads.
[ 2813.040289]   <idle>-0       4d.h. 2578022868us : check_preempt_curr: entered
[ 2813.040289]   <idle>-0       4d.h. 2578022868us : resched_curr: entered
[ 2813.040289] torture_-135     0.N.. 2578022872us : cpu_stop_queue_work: entered for CPU 6
[ 2813.040289] torture_-135     0.N.. 2578022873us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022873us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022874us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-125     5d... 2578022874us : scheduler_ipi: entered
	CPU 5 got its IPI, so CPUs 1-5 IPIed and CPUs 1-5 received IPI.
[ 2813.040289] rcu_tort-125     5d.h. 2578022874us : sched_ttwu_pending: entered
[ 2813.040289] rcu_tort-125     5d.h. 2578022875us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] rcu_tort-125     5d.h. 2578022875us : ttwu_do_activate.isra.108: entered
[ 2813.040289] rcu_tort-125     5d.h. 2578022875us : activate_task: entered
[ 2813.040289] rcu_tort-125     5d.h. 2578022877us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-125     5d.h. 2578022877us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-125     5d.h. 2578022877us : resched_curr: entered
[ 2813.040289] rcu_tort-128     3d... 2578022877us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=R+ ==> next_comm=migration/3 next_pid=27 next_prio=0
	CPUs 1-3 have switched to their migration kthreads.
[ 2813.040289]   <idle>-0       4.N.. 2578022912us : sched_ttwu_pending: entered
[ 2813.040289] torture_-135     0dN.. 2578022914us : try_to_wake_up: entered
[ 2813.040289]   <idle>-0       4d... 2578022914us : sched_switch: prev_comm=swapper/4 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=migration/4 next_pid=33 next_prio=0
	CPUs 1-4 have switched to their migration kthreads.
[ 2813.040289] torture_-135     0dN.. 2578022915us : try_to_wake_up: entered, CPU 6
	We have IPIed CPUs 1-6.
[ 2813.040289] rcu_tort-125     5d... 2578022926us : sched_switch: prev_comm=rcu_torture_rea prev_pid=125 prev_prio=139 prev_state=R+ ==> next_comm=migration/5 next_pid=40 next_prio=0
	CPUs 1-5 have switched to their migration kthreads.
[ 2813.040289] rcu_tort-136     6d... 2578022926us : scheduler_ipi: entered
	CPU 6 got its IPI, so CPUs 1-6 IPIed and CPUs 1-6 received IPI.
[ 2813.040289] torture_-135     0.N.. 2578022929us : cpu_stop_queue_work: entered for CPU 7
[ 2813.040289] torture_-135     0.N.. 2578022930us : wake_up_q: entered
[ 2813.040289] torture_-135     0.N.. 2578022930us : wake_up_process: entered
[ 2813.040289] torture_-135     0.N.. 2578022930us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-136     6d.h. 2578022930us : sched_ttwu_pending: entered
[ 2813.040289] torture_-135     0dN.. 2578022930us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-136     6d.h. 2578022930us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] rcu_tort-136     6d.h. 2578022930us : ttwu_do_activate.isra.108: entered
[ 2813.040289] torture_-135     0dN.. 2578022931us : try_to_wake_up: entered, CPU 7
	We have IPIed CPUs 1-7.
[ 2813.040289] rcu_tort-136     6d.h. 2578022931us : activate_task: entered
[ 2813.040289] rcu_tort-136     6d.h. 2578022931us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-136     6d.h. 2578022931us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-136     6d.h. 2578022931us : resched_curr: entered
[ 2813.040289] torture_-135     0d... 2578022951us : sched_switch: prev_comm=torture_onoff prev_pid=135 prev_prio=120 prev_state=D ==> next_comm=migration/0 next_pid=11 next_prio=0
	CPUs 0-5 have switched to their migration kthreads.
[ 2813.040289] rcu_tort-136     6d... 2578022951us : sched_switch: prev_comm=rcu_torture_fwd prev_pid=136 prev_prio=139 prev_state=R+ ==> next_comm=migration/6 next_pid=46 next_prio=0
	CPUs 0-6 have switched to their migration kthreads.
[ 2813.040289] migratio-46      6d.s1 2578023060us : wake_up_process: entered
[ 2813.040289] migratio-46      6d.s1 2578023060us : try_to_wake_up: entered
[ 2813.040289] migratio-46      6d.s1 2578023060us : try_to_wake_up: entered
[ 2813.040289] migratio-46      6d.s1 2578023061us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-46      6d.s1 2578023061us : activate_task: entered
[ 2813.040289] migratio-46      6d.s1 2578023062us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-46      6d.s1 2578023062us : check_preempt_curr: entered
[ 2813.040289]   <idle>-0       7d... 2578023758us : scheduler_ipi: entered
	CPU 7 got its IPI, so CPUs 1-7 IPIed and CPUs 1-7 received IPI.
[ 2813.040289]   <idle>-0       7d.h. 2578025037us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       7d.h. 2578025038us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]   <idle>-0       7d.h. 2578025038us : ttwu_do_activate.isra.108: entered
[ 2813.040289]   <idle>-0       7d.h. 2578025038us : activate_task: entered
[ 2813.040289]   <idle>-0       7d.h. 2578025043us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289]   <idle>-0       7d.h. 2578025043us : check_preempt_curr: entered
[ 2813.040289]   <idle>-0       7d.h. 2578025043us : resched_curr: entered
[ 2813.040289]   <idle>-0       7.N.. 2578025107us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       7d... 2578025132us : sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=migration/7 next_pid=52 next_prio=0
	CPUs 0-7 have all switched to their migration kthreads.  So we
	should be done, but we clearly are not.  Almost two minutes later:
[ 2813.040289] migratio-33      4d..1 2689243033us : sched_ttwu_pending: entered
[ 2813.040289] migratio-11      0d.h1 2689243195us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.h1 2689243196us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.h1 2689243198us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.h1 2689243201us : try_to_wake_up: entered, CPU 5
[ 2813.040289] migratio-40      5d... 2689243204us : sched_switch: prev_comm=migration/5 prev_pid=40 prev_prio=0 prev_state=S ==> next_comm=rcu_torture_rea next_pid=125 next_prio=139
	CPU 5 switches from its migration kthread.
[ 2813.040289] migratio-46      6d... 2689243222us : sched_switch: prev_comm=migration/6 prev_pid=46 prev_prio=0 prev_state=S ==> next_comm=kworker/6:1 next_pid=175 next_prio=120
	CPU 6 switches from its migration kthread.
[ 2813.040289] migratio-21      2d.s1 2689243222us : activate_task: entered
[ 2813.040289] migratio-21      2d.s1 2689243225us : check_preempt_curr: entered
[ 2813.040289] migratio-27      3d.s1 2689243226us : activate_task: entered
[ 2813.040289] rcu_tort-125     5d... 2689243227us : scheduler_ipi: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243228us : sched_ttwu_pending: entered
[ 2813.040289] migratio-27      3d.s1 2689243228us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243228us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] migratio-27      3d.s1 2689243229us : activate_task: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243229us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0..s1 2689243229us : wake_up_process: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243229us : activate_task: entered
[ 2813.040289] migratio-27      3d.s1 2689243229us : check_preempt_curr: entered
[ 2813.040289] migratio-11      0..s1 2689243229us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243230us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243231us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243231us : activate_task: entered
[ 2813.040289] migratio-21      2d... 2689243232us : sched_switch: prev_comm=migration/2 prev_pid=21 prev_prio=0 prev_state=S ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
	CPU 2 switches from its migration kthread.
[ 2813.040289] migratio-11      0d.s1 2689243232us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243232us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243232us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243232us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-125     5d.h. 2689243233us : resched_curr: entered
[ 2813.040289] migratio-11      0..s1 2689243234us : wake_up_process: entered
[ 2813.040289] migratio-11      0..s1 2689243234us : try_to_wake_up: entered
[ 2813.040289] migratio-27      3d... 2689243234us : sched_switch: prev_comm=migration/3 prev_pid=27 prev_prio=0 prev_state=S ==> next_comm=rcu_torture_fak next_pid=122 next_prio=139
	CPU 3 switches from its migration kthread.
[ 2813.040289] migratio-11      0d.s1 2689243235us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243235us : ttwu_do_activate.isra.108: entered
[ 2813.040289] kworker/-175     6d... 2689243235us : sched_switch: prev_comm=kworker/6:1 prev_pid=175 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_fwd next_pid=136 next_prio=139
	CPU 6 switches yet again.
[ 2813.040289] migratio-11      0d.s1 2689243235us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243236us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243237us : check_preempt_curr: entered
[ 2813.040289] migratio-11      0d.s1 2689243239us : wake_up_process: entered
[ 2813.040289] rcu_tort-126     2d... 2689243240us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
	CPU 2 switches yet again.
[ 2813.040289] migratio-11      0d.s1 2689243240us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243240us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243240us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243240us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243241us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243241us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-125     5d... 2689243242us : sched_switch: prev_comm=rcu_torture_rea prev_pid=125 prev_prio=139 prev_state=S ==> next_comm=init next_pid=1 next_prio=120
	CPU 5 switches yet again.
[ 2813.040289] migratio-11      0d.s1 2689243242us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243243us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243243us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243243us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243244us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243244us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-124     2d... 2689243244us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
	CPU 2 switches yet again.
[ 2813.040289] migratio-11      0d.s1 2689243245us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243245us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243245us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243246us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243246us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243246us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0..s1 2689243247us : wake_up_process: entered
[ 2813.040289] migratio-11      0..s1 2689243247us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243248us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243248us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243249us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243249us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243249us : check_preempt_curr: entered
[ 2813.040289] migratio-11      0d.s1 2689243251us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243251us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243251us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243252us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243252us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243252us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243253us : check_preempt_curr: entered
[ 2813.040289] migratio-11      0d.s1 2689243263us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243263us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243264us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243264us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0..s1 2689243265us : wake_up_process: entered
[ 2813.040289] migratio-11      0..s1 2689243265us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243266us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243266us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243266us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243268us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243268us : check_preempt_curr: entered
[ 2813.040289] migratio-11      0..s1 2689243269us : wake_up_process: entered
[ 2813.040289] migratio-11      0..s1 2689243269us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243270us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243270us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-11      0d.s1 2689243270us : activate_task: entered
[ 2813.040289] migratio-11      0d.s1 2689243271us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-11      0d.s1 2689243272us : check_preempt_curr: entered
[ 2813.040289] migratio-52      7d... 2689243291us : sched_switch: prev_comm=migration/7 prev_pid=52 prev_prio=0 prev_state=S ==> next_comm=swapper/7 next_pid=0 next_prio=120
	CPU 7 switches from its migration kthread.
[ 2813.040289] migratio-11      0d.s1 2689243315us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243315us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243316us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243316us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243316us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243317us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d.s1 2689243317us : wake_up_process: entered
[ 2813.040289] migratio-11      0d.s1 2689243318us : try_to_wake_up: entered
[ 2813.040289] migratio-11      0d... 2689243323us : sched_switch: prev_comm=migration/0 prev_pid=11 prev_prio=0 prev_state=S ==> next_comm=kworker/0:3 next_pid=171 next_prio=120
	CPU 0 switches from its migration kthread.
[ 2813.040289] kworker/-171     0d... 2689243327us : try_to_wake_up: entered
[ 2813.040289] kworker/-171     0d... 2689243327us : try_to_wake_up: entered
[ 2813.040289] kworker/-171     0d... 2689243328us : try_to_wake_up: entered, CPU 5
[ 2813.040289]     init-1       5d... 2689243343us : scheduler_ipi: entered
[ 2813.040289]     init-1       5d.h. 2689243344us : sched_ttwu_pending: entered
[ 2813.040289]     init-1       5d.h. 2689243344us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]     init-1       5d.h. 2689243351us : ttwu_do_activate.isra.108: entered
[ 2813.040289]     init-1       5d.h. 2689243351us : activate_task: entered
[ 2813.040289]     init-1       5d.h. 2689243353us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289]     init-1       5d.h. 2689243353us : check_preempt_curr: entered
[ 2813.040289] kworker/-171     0d... 2689243362us : sched_switch: prev_comm=kworker/0:3 prev_pid=171 prev_prio=120 prev_state=I ==> next_comm=rcu_sched next_pid=10 next_prio=120
	CPU 0 switches again.
[ 2813.040289] rcu_sche-10      0d... 2689243365us : try_to_wake_up: entered
[ 2813.040289] rcu_sche-10      0d... 2689243366us : try_to_wake_up: entered
[ 2813.040289] rcu_sche-10      0d... 2689243366us : try_to_wake_up: entered, CPU 5
[ 2813.040289] rcu_sche-10      0d... 2689243379us : sched_switch: prev_comm=rcu_sched prev_pid=10 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_fak next_pid=123 next_prio=139
	CPU 0 switches again.
[ 2813.040289]     init-1       5d... 2689243380us : scheduler_ipi: entered
[ 2813.040289]     init-1       5d.h. 2689243380us : sched_ttwu_pending: entered
[ 2813.040289]     init-1       5d.h. 2689243380us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]     init-1       5d.h. 2689243380us : ttwu_do_activate.isra.108: entered
[ 2813.040289]     init-1       5d.h. 2689243381us : activate_task: entered
[ 2813.040289]     init-1       5d.h. 2689243382us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289]     init-1       5d.h. 2689243382us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-123     0d... 2689243509us : sched_switch: prev_comm=rcu_torture_fak prev_pid=123 prev_prio=139 prev_state=D ==> next_comm=kworker/u16:0 next_pid=181 next_prio=120
	CPU 0 switches again.
[ 2813.040289] kworker/-181     0d... 2689243510us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243510us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243510us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243511us : ttwu_do_activate.isra.108: entered
[ 2813.040289] kworker/-181     0d... 2689243511us : activate_task: entered
[ 2813.040289] kworker/-181     0d... 2689243512us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] kworker/-181     0d... 2689243512us : check_preempt_curr: entered
[ 2813.040289] kworker/-181     0d... 2689243513us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243513us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243513us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243513us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243514us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243514us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243514us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243514us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243515us : wake_up_process: entered
[ 2813.040289] kworker/-181     0d... 2689243515us : try_to_wake_up: entered
[ 2813.040289] kworker/-181     0d... 2689243516us : sched_switch: prev_comm=kworker/u16:0 prev_pid=181 prev_prio=120 prev_state=I ==> next_comm=torture_shuffle next_pid=132 next_prio=120
	CPU 0 switches again.
[ 2813.040289] torture_-132     0.... 2689243518us : wake_up_q: entered
[ 2813.040289] torture_-132     0d... 2689243519us : sched_switch: prev_comm=torture_shuffle prev_pid=132 prev_prio=120 prev_state=D ==> next_comm=kworker/0:1 next_pid=183 next_prio=120
	CPU 0 switches again.
[ 2813.040289] kworker/-183     0d... 2689243521us : sched_switch: prev_comm=kworker/0:1 prev_pid=183 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_sta next_pid=131 next_prio=120
	CPU 0 switches again.
[ 2813.040289] rcu_tort-131     0d... 2689243559us : sched_switch: prev_comm=rcu_torture_sta prev_pid=131 prev_prio=120 prev_state=S ==> next_comm=torture_stutter next_pid=133 next_prio=120
	CPU 0 switches again.
[ 2813.040289] torture_-133     0d... 2689243561us : sched_switch: prev_comm=torture_stutter prev_pid=133 prev_prio=120 prev_state=S ==> next_comm=kworker/u16:4 next_pid=161 next_prio=120
[ 2813.040289] kworker/-161     0d... 2689243566us : activate_task: entered
[ 2813.040289] kworker/-161     0d... 2689243567us : check_preempt_curr: entered
[ 2813.040289] kworker/-161     0d... 2689243568us : sched_switch: prev_comm=kworker/u16:4 prev_pid=161 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_rea next_pid=130 next_prio=139
	CPU 0 switches again.
[ 2813.040289] rcu_tort-130     0d... 2689243571us : activate_task: entered
[ 2813.040289] rcu_tort-130     0d... 2689243572us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-130     0d... 2689243572us : activate_task: entered
[ 2813.040289] rcu_tort-130     0d... 2689243572us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-130     0d... 2689243573us : sched_switch: prev_comm=rcu_torture_rea prev_pid=130 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=127 next_prio=139
	CPU 0 switches again.
[ 2813.040289] rcu_tort-127     0d... 2689243585us : sched_switch: prev_comm=rcu_torture_rea prev_pid=127 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=128 next_prio=139
	CPU 0 switches again.
[ 2813.040289] rcu_tort-128     0d... 2689243588us : activate_task: entered
[ 2813.040289] rcu_tort-128     0d... 2689243588us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-128     0d... 2689243588us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=129 next_prio=139
	CPU 0 switches again.
[ 2813.040289] rcu_tort-129     0d... 2689243591us : sched_switch: prev_comm=rcu_torture_rea prev_pid=129 prev_prio=139 prev_state=S ==> next_comm=swapper/0 next_pid=0 next_prio=120
	CPU 0 switches again.
[ 2813.040289] rcu_tort-122     3.... 2689243707us : wake_up_process: entered
[ 2813.040289] rcu_tort-122     3.... 2689243707us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-122     3d... 2689243711us : sched_switch: prev_comm=rcu_torture_fak prev_pid=122 prev_prio=139 prev_state=D ==> next_comm=swapper/3 next_pid=0 next_prio=120
	CPU 3 switches again.
[ 2813.040289]   <idle>-0       0d... 2689244073us : wake_up_process: entered
[ 2813.040289]   <idle>-0       0d... 2689244074us : try_to_wake_up: entered
[ 2813.040289]   <idle>-0       0d... 2689244074us : try_to_wake_up: entered
[ 2813.040289]   <idle>-0       0d... 2689244074us : ttwu_do_activate.isra.108: entered
[ 2813.040289]   <idle>-0       0d... 2689244075us : activate_task: entered
[ 2813.040289]   <idle>-0       0d... 2689244076us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289]   <idle>-0       0d... 2689244076us : check_preempt_curr: entered
[ 2813.040289]   <idle>-0       0d... 2689244076us : resched_curr: entered
[ 2813.040289]   <idle>-0       0.N.. 2689244087us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       0d... 2689244088us : sched_switch: prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ksoftirqd/0 next_pid=9 next_prio=120
	CPU 0 switches again.
[ 2813.040289]     init-1       5d... 2689244336us : sched_switch: prev_comm=init prev_pid=1 prev_prio=120 prev_state=S ==> next_comm=rcu_torture_fak next_pid=120 next_prio=139
	CPU 5 switches again.
[ 2813.040289] rcu_tort-120     5.... 2689244344us : wake_up_q: entered
[ 2813.040289] rcu_tort-120     5.... 2689244344us : wake_up_process: entered
[ 2813.040289] rcu_tort-120     5.... 2689244344us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-120     5d... 2689244345us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-120     5d... 2689244346us : try_to_wake_up: entered, CPU 0
[ 2813.040289] rcu_tort-120     5d... 2689244359us : sched_switch: prev_comm=rcu_torture_fak prev_pid=120 prev_prio=139 prev_state=S ==> next_comm=rcuog/1 next_pid=18 next_prio=120
	CPU 5 switches again.
[ 2813.040289]  rcuog/1-18      5d... 2689244361us : wake_up_process: entered
[ 2813.040289]  rcuog/1-18      5d... 2689244362us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689244362us : scheduler_ipi: entered
[ 2813.040289]  rcuog/1-18      5d... 2689244362us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244362us : sched_ttwu_pending: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244363us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]  rcuog/1-18      5d... 2689244363us : try_to_wake_up: entered, CPU 0
[ 2813.040289] ksoftirq-9       0d.H. 2689244363us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244363us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244365us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244365us : check_preempt_curr: entered
[ 2813.040289]  rcuog/1-18      5d... 2689244375us : sched_switch: prev_comm=rcuog/1 prev_pid=18 prev_prio=120 prev_state=S ==> next_comm=swapper/5 next_pid=0 next_prio=120
	CPU 5 switches again.
[ 2813.040289] ksoftirq-9       0d.s. 2689244386us : scheduler_ipi: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244386us : sched_ttwu_pending: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244387us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] ksoftirq-9       0d.H. 2689244387us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244387us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244388us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689244388us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245078us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245078us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245079us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245079us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245080us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245080us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245080us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245081us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245081us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245081us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245082us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245082us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245082us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245083us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245083us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245083us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245084us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245084us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245084us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245084us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245085us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245085us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689245085us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245086us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245086us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245086us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245087us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689245087us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-136     6.... 2689245087us : wake_up_process: entered
[ 2813.040289] rcu_tort-136     6.... 2689245087us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-136     6d... 2689245087us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-136     6d... 2689245087us : try_to_wake_up: entered, CPU 0
[ 2813.040289] ksoftirq-9       0d.s. 2689245639us : scheduler_ipi: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245639us : sched_ttwu_pending: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245639us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] ksoftirq-9       0d.H. 2689245639us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245640us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245640us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245640us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245640us : resched_curr: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : activate_task: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : activate_task: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : resched_curr: entered
[ 2813.040289] rcu_tort-136     6dN.. 2689245640us : activate_task: entered
[ 2813.040289] rcu_tort-136     6dN.. 2689245640us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-136     6d... 2689245640us : sched_switch: prev_comm=rcu_torture_fwd prev_pid=136 prev_prio=139 prev_state=D ==> next_comm=rcuos/2 next_pid=25 next_prio=120
	CPU 6 switches again.
[ 2813.040289] ksoftirq-9       0d... 2689245654us : sched_switch: prev_comm=ksoftirqd/0 prev_pid=9 prev_prio=120 prev_state=R+ ==> next_comm=rcuog/4 next_pid=37 next_prio=120
	CPU 0 switches again.
[ 2813.040289]  rcuog/4-37      0d... 2689245658us : sched_switch: prev_comm=rcuog/4 prev_pid=37 prev_prio=120 prev_state=S ==> next_comm=ksoftirqd/0 next_pid=9 next_prio=120
	CPU 0 switches again.
[ 2813.040289]  rcuos/2-25      6d... 2689245658us : try_to_wake_up: entered
[ 2813.040289]  rcuos/2-25      6d... 2689245658us : try_to_wake_up: entered
[ 2813.040289]  rcuos/2-25      6d... 2689245658us : try_to_wake_up: entered, CPU 2
[ 2813.040289]  rcuos/2-25      6d... 2689245658us : sched_switch: prev_comm=rcuos/2 prev_pid=25 prev_prio=120 prev_state=S ==> next_comm=rcu_torture_fak next_pid=123 next_prio=139
	CPU 6 switches again.
[ 2813.040289] rcu_tort-123     6d... 2689245658us : wake_up_process: entered
[ 2813.040289] rcu_tort-123     6d... 2689245658us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-123     6d... 2689245658us : try_to_wake_up: entered
[ 2813.040289] rcu_tort-123     6d... 2689245658us : try_to_wake_up: entered, CPU 0
[ 2813.040289] rcu_tort-123     6d... 2689245658us : sched_switch: prev_comm=rcu_torture_fak prev_pid=123 prev_prio=139 prev_state=D ==> next_comm=rcu_torture_rea next_pid=129 next_prio=139
	CPU 6 switches again.
[ 2813.040289] rcu_tort-129     6d... 2689245658us : activate_task: entered
[ 2813.040289] rcu_tort-129     6d... 2689245658us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-129     6d... 2689245658us : sched_switch: prev_comm=rcu_torture_rea prev_pid=129 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=128 next_prio=139
	CPU 6 switches again.
[ 2813.040289] ksoftirq-9       0d.s. 2689245781us : scheduler_ipi: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245781us : sched_ttwu_pending: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245781us : sched_ttwu_pending: non-NULL llist
[ 2813.040289] ksoftirq-9       0d.H. 2689245781us : ttwu_do_activate.isra.108: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245782us : activate_task: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245782us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245782us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0d.H. 2689245783us : resched_curr: entered
[ 2813.040289] rcu_tort-128     6d... 2689245783us : activate_task: entered
[ 2813.040289] rcu_tort-128     6d... 2689245783us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-128     6d... 2689245783us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=127 next_prio=139
	CPU 6 switches again.
[ 2813.040289] rcu_tort-127     6d... 2689245783us : activate_task: entered
[ 2813.040289] rcu_tort-127     6d... 2689245783us : check_preempt_curr: entered
[ 2813.040289] rcu_tort-127     6d... 2689245783us : sched_switch: prev_comm=rcu_torture_rea prev_pid=127 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=130 next_prio=139
	CPU 6 switches again.
[ 2813.040289] rcu_tort-130     6d... 2689245783us : sched_switch: prev_comm=rcu_torture_rea prev_pid=130 prev_prio=139 prev_state=S ==> next_comm=swapper/6 next_pid=0 next_prio=120
	CPU 6 switches again.
[ 2813.040289] ksoftirq-9       0d... 2689245805us : sched_switch: prev_comm=ksoftirqd/0 prev_pid=9 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:1 next_pid=183 next_prio=120
	CPU 0 switches again.
[ 2813.040289]   <idle>-0       2d... 2689245805us : scheduler_ipi: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246047us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246047us : sched_ttwu_pending: non-NULL llist
[ 2813.040289]   <idle>-0       2d.h. 2689246048us : ttwu_do_activate.isra.108: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246048us : activate_task: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246050us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246050us : check_preempt_curr: entered
[ 2813.040289]   <idle>-0       2d.h. 2689246050us : resched_curr: entered
[ 2813.040289]   <idle>-0       2.N.. 2689246090us : sched_ttwu_pending: entered
[ 2813.040289]   <idle>-0       2d... 2689246091us : sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=rcu_torture_wri next_pid=119 next_prio=120
	CPU 2 switches again.
[ 2813.040289] rcu_tort-119     2d... 2689246096us : sched_switch: prev_comm=rcu_torture_wri prev_pid=119 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
	CPU 2 switches again.
[ 2813.040289] kworker/-183     0d.h. 2689280510us : resched_curr: entered
[ 2813.040289] kworker/-183     0dNh. 2689285096us : resched_curr: entered
[ 2813.040289] kworker/-183     0dN.. 2689285103us : wake_up_process: entered
[ 2813.040289] kworker/-183     0dN.. 2689285103us : try_to_wake_up: entered
[ 2813.040289] kworker/-183     0dN.. 2689285104us : try_to_wake_up: entered
[ 2813.040289] kworker/-183     0dN.. 2689285105us : ttwu_do_activate.isra.108: entered
[ 2813.040289] kworker/-183     0dN.. 2689285105us : activate_task: entered
[ 2813.040289] kworker/-183     0dN.. 2689285106us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] kworker/-183     0dN.. 2689285106us : check_preempt_curr: entered
[ 2813.040289] kworker/-183     0d... 2689285109us : sched_switch: prev_comm=kworker/0:1 prev_pid=183 prev_prio=120 prev_state=D ==> next_comm=kworker/0:3 next_pid=171 next_prio=120
	CPU 0 switches again.
[ 2813.040289] kworker/-171     0d... 2689285121us : try_to_wake_up: entered
[ 2813.040289] kworker/-171     0d... 2689285121us : try_to_wake_up: entered
[ 2813.040289] kworker/-171     0d... 2689285122us : ttwu_do_activate.isra.108: entered
[ 2813.040289] kworker/-171     0d... 2689285122us : activate_task: entered
[ 2813.040289] kworker/-171     0d... 2689285122us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] kworker/-171     0d... 2689285122us : check_preempt_curr: entered
[ 2813.040289] kworker/-171     0d... 2689285123us : sched_switch: prev_comm=kworker/0:3 prev_pid=171 prev_prio=120 prev_state=I ==> next_comm=ksoftirqd/0 next_pid=9 next_prio=120
	CPU 0 switches again.
[ 2813.040289] ksoftirq-9       0..s. 2689285125us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285126us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285126us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285127us : try_to_wake_up: entered, CPU 5
[ 2813.040289] ksoftirq-9       0..s. 2689285138us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285138us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285138us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285138us : try_to_wake_up: entered, CPU 6
[ 2813.040289] migratio-14      1dN.1 2689285146us : try_to_wake_up: entered
[ 2813.040289] migratio-14      1dN.1 2689285148us : try_to_wake_up: entered
[ 2813.040289] migratio-14      1dN.1 2689285148us : ttwu_do_activate.isra.108: entered
[ 2813.040289] migratio-14      1dN.1 2689285148us : activate_task: entered
[ 2813.040289] migratio-14      1dN.1 2689285149us : ttwu_do_wakeup.isra.107: entered
[ 2813.040289] migratio-14      1dN.1 2689285149us : check_preempt_curr: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285150us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285150us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285150us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285151us : try_to_wake_up: entered, CPU 6
[ 2813.040289] ksoftirq-9       0..s. 2689285151us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285151us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285152us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285152us : try_to_wake_up: entered, CPU 6
[ 2813.040289] ksoftirq-9       0..s. 2689285152us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285152us : try_to_wake_up: entered
[ 2813.040289] migratio-14      1d... 2689285153us : sched_switch: prev_comm=migration/1 prev_pid=14 prev_prio=0 prev_state=S ==> next_comm=torture_onoff next_pid=135 next_prio=120
	CPU 1 switches from its migration kthread.
[ 2813.040289] ksoftirq-9       0d.s. 2689285153us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285153us : try_to_wake_up: entered, CPU 6
[ 2813.040289] ksoftirq-9       0..s. 2689285153us : wake_up_process: entered
[ 2813.040289] ksoftirq-9       0..s. 2689285153us : try_to_wake_up: entered
[ 2813.040289] ksoftirq-9       0d.s. 2689285154us : try_to_wake_up: entered
[ 2813.040289] ---------------------------------

And finally, the trace ends and CPU 4 is announced as being offline:

[ 2813.040289] smpboot: CPU 4 is now offline
#39 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 42 lines · Paul E. McKenney
Quoted 27 lines · Paul E. McKenney
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 7 lines
Quoted 2 lines

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

And I did eventually get a good trace. If I am interpreting this trace correctly, the torture_-135 task didn't get around to attempting to wake up all of the CPUs. I will try again, but this time with the sched_switch trace event enabled.

Quoted 7 lines

As a side note, enabling ftrace from the command line seems to interact badly with turning tracing off and on in the kernel, so I eventually resorted to trace_printk() in the functions of interest. The trace output is below, followed by the current diagnostic patch. Please note that I am -not- using the desperation hammer-the-scheduler patches.

More as I learn more!

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

Quoted 7 lines

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Functionally, multi_cpu_stop() is working fine, according to the trace below (search for a line beginning with TAB). But somehow CPU 2 took almost three -minutes- to do one iteration of the loop. The prime suspect in that loop is cpu_relax() due to the hypervisor having an opportunity to do something at that point. The commentary below (again, search for a line beginning with TAB) gives my analysis.

Of course, if I am correct, it should be possible to catch cpu_relax() in the act. That is the next step, give or take the Heisenbuggy nature of this beast.

Another thing for me to try is to run longer with !NO_HZ_FULL, just in case the earlier runs just got lucky.

Thoughts?

Thanx, Paul

------------------------------------------------------------------------

[ 1564.195213] Unregister pv shared memory for cpu 6
[ 1731.578001] rcu: INFO: rcu_sched self-detected stall on CPU
...
[ 1731.632619] torture_-135     0.... 1562233456us : __stop_cpus: CPUs 0-2,6-7 online
[ 1731.632619] torture_-135     0.... 1562233457us : queue_stop_cpus_work: entered
[ 1731.632619] torture_-135     0.... 1562233457us : cpu_stop_queue_work: entered for CPU 0
[ 1731.632619] torture_-135     0.... 1562233458us : wake_up_q: entered
[ 1731.632619] torture_-135     0.... 1562233458us : wake_up_process: entered
[ 1731.632619] torture_-135     0.... 1562233458us : try_to_wake_up: entered
[ 1731.632619] torture_-135     0d... 1562233458us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] torture_-135     0d... 1562233459us : ttwu_do_activate.isra.108: entered
[ 1731.632619] torture_-135     0d... 1562233459us : activate_task: entered
[ 1731.632619] torture_-135     0d... 1562233459us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] torture_-135     0d... 1562233459us : check_preempt_curr: entered
[ 1731.632619] torture_-135     0d... 1562233459us : resched_curr: entered
[ 1731.632619] torture_-135     0.N.. 1562233460us : cpu_stop_queue_work: entered for CPU 1
[ 1731.632619] torture_-135     0.N.. 1562233460us : wake_up_q: entered
[ 1731.632619] torture_-135     0.N.. 1562233460us : wake_up_process: entered
[ 1731.632619] torture_-135     0.N.. 1562233460us : try_to_wake_up: entered
[ 1731.632619] torture_-135     0dN.. 1562233461us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] torture_-135     0dN.. 1562233478us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619] torture_-135     0.N.. 1562233488us : cpu_stop_queue_work: entered for CPU 2
[ 1731.632619] torture_-135     0.N.. 1562233488us : wake_up_q: entered
[ 1731.632619] torture_-135     0.N.. 1562233488us : wake_up_process: entered
[ 1731.632619] torture_-135     0.N.. 1562233488us : try_to_wake_up: entered
[ 1731.632619] torture_-135     0dN.. 1562233489us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] torture_-135     0dN.. 1562233489us : try_to_wake_up: ttwu_queue_remote entered, CPU 2
[ 1731.632619]   <idle>-0       1d... 1562233493us : scheduler_ipi: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233495us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233495us : sched_ttwu_pending: non-NULL llist
[ 1731.632619]   <idle>-0       1d.h. 1562233495us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233495us : activate_task: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233496us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233496us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       1d.h. 1562233496us : resched_curr: entered
[ 1731.632619] torture_-135     0.N.. 1562233498us : cpu_stop_queue_work: entered for CPU 6
[ 1731.632619] torture_-135     0.N.. 1562233498us : wake_up_q: entered
[ 1731.632619] torture_-135     0.N.. 1562233499us : wake_up_process: entered
[ 1731.632619] torture_-135     0.N.. 1562233499us : try_to_wake_up: entered
[ 1731.632619] torture_-135     0dN.. 1562233499us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] torture_-135     0dN.. 1562233499us : try_to_wake_up: ttwu_queue_remote entered, CPU 6
[ 1731.632619] torture_-135     0.N.. 1562233509us : cpu_stop_queue_work: entered for CPU 7
[ 1731.632619] torture_-135     0.N.. 1562233509us : wake_up_q: entered
[ 1731.632619] torture_-135     0.N.. 1562233509us : wake_up_process: entered
[ 1731.632619] torture_-135     0.N.. 1562233509us : try_to_wake_up: entered
[ 1731.632619] torture_-135     0dN.. 1562233510us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] torture_-135     0dN.. 1562233510us : try_to_wake_up: ttwu_queue_remote entered, CPU 7
[ 1731.632619]   <idle>-0       2d... 1562233515us : scheduler_ipi: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233517us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233517us : sched_ttwu_pending: non-NULL llist
[ 1731.632619]   <idle>-0       2d.h. 1562233518us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233519us : activate_task: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233519us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233519us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       2d.h. 1562233519us : resched_curr: entered
[ 1731.632619] torture_-135     0d... 1562233520us : sched_switch: prev_comm=torture_onoff prev_pid=135 prev_prio=120 prev_state=D ==> next_comm=migration/0 next_pid=11 next_prio=0
[ 1731.632619] rcu_tort-128     7d... 1562233520us : scheduler_ipi: entered
[ 1731.632619] rcu_tort-128     7d.h. 1562233521us : sched_ttwu_pending: entered
[ 1731.632619] rcu_tort-128     7d.h. 1562233521us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] rcu_tort-128     7d.h. 1562233521us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-128     7d.h. 1562233521us : activate_task: entered
[ 1731.632619] migratio-11      0...1 1562233521us : multi_cpu_stop: curstate = 1, ack = 5
[ 1731.632619] rcu_tort-128     7d.h. 1562233521us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-128     7d.h. 1562233523us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-128     7d.h. 1562233524us : resched_curr: entered
[ 1731.632619] rcu_tort-128     7d... 1562233532us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=R+ ==> next_comm=migration/7 next_pid=52 next_prio=0
[ 1731.632619] migratio-52      7...1 1562233535us : multi_cpu_stop: curstate = 1, ack = 4
[ 1731.632619]   <idle>-0       1dNs. 1562233552us : activate_task: entered
[ 1731.632619]   <idle>-0       2.N.. 1562233553us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       1dNs. 1562233553us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       2d... 1562233554us : sched_switch: prev_comm=swapper/2 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=migration/2 next_pid=21 next_prio=0
[ 1731.632619]   <idle>-0       1dNs. 1562233554us : resched_curr: entered
[ 1731.632619]   <idle>-0       1.N.. 1562233554us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       1d... 1562233555us : sched_switch: prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=migration/1 next_pid=14 next_prio=0
[ 1731.632619] migratio-21      2...1 1562233556us : multi_cpu_stop: curstate = 1, ack = 3
[ 1731.632619] migratio-14      1...1 1562233556us : multi_cpu_stop: curstate = 1, ack = 2
[ 1731.632619] migratio-11      0..s1 1562234528us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1562234528us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1562234529us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1562234529us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1562234529us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1562234529us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1562234530us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0..s1 1562235527us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1562235527us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1562235528us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1562235528us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1562235528us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1562235528us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1562235529us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       6d... 1562235529us : scheduler_ipi: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249376us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249377us : sched_ttwu_pending: non-NULL llist
[ 1731.632619]   <idle>-0       6d.h. 1562249377us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249377us : activate_task: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249378us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249378us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       6d.h. 1562249378us : resched_curr: entered
[ 1731.632619]   <idle>-0       6.N.. 1562249472us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       6d... 1562249480us : sched_switch: prev_comm=swapper/6 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=migration/6 next_pid=46 next_prio=0
[ 1731.632619] migratio-46      6...1 1562249502us : multi_cpu_stop: curstate = 1, ack = 1
[ 1731.632619] migratio-14      1...1 1562249503us : multi_cpu_stop: curstate = 2, ack = 5
[ 1731.632619] migratio-52      7...1 1562249503us : multi_cpu_stop: curstate = 2, ack = 5
[ 1731.632619] migratio-11      0...1 1562249504us : multi_cpu_stop: curstate = 2, ack = 3
[ 1731.632619] migratio-46      6...1 1562249505us : multi_cpu_stop: curstate = 2, ack = 2
	Almost three -minutes- delay.  CPU 2 was just fine 16ms earlier:
	2...1 1562233556us : multi_cpu_stop: curstate = 1, ack = 3
	"curstate = 1" above corresponds to MULTI_STOP_PREPARE.
	"curstate = 2" below corresponds to MULTI_STOP_DISABLE_IRQ
	So what did that CPU need to execute in the meantime?
	o	ack_state(), which does an atomic_dec_and_test(), an
		atomic_set(), an smp_wmb(), and a WRITE_ONCE().
	o	rcu_momentary_dyntick_idle(), which does a
		raw_cpu_write(), an atomic_add_return(),
		integer comparison for a WARN_ON_ONCE() and call to
		rcu_preempt_deferred_qs(), which in this configuration
		is an empty function.
	o	stop_machine_yield(), which does a cpu_relax(), which might
		pass control to the hupervisor.
	o	Multiple passes through a loop doing READ_ONCE(msdata->state)
		(paranoia on my part), a couple of comparisons, an
		rcu_momentary_dyntick_idle(), and a stop_machine_yield().
	o	We don't call the function msdata->fn(msdata->data)
		because that doesn't happen until after we get to
		MULTI_STOP_RUN, which is after MULTI_STOP_DISABLE_IRQ.
	o	This would be easy to blame on the hypervisor, but then
		why is this behavior restricted to CONFIG_NO_HZ_FULL
		guest-OS kernels with lots of nohz_full CPUs?  I suppose
		maybe the guest's scheduling-clock interrupt attempts
		might be having an effect, but...
[ 1731.632619] migratio-21      2...1 1729613677us : multi_cpu_stop: curstate = 2, ack = 1
[ 1731.632619] migratio-21      2d..1 1729613680us : multi_cpu_stop: curstate = 3, ack = 5
[ 1731.632619] migratio-46      6d..1 1729613680us : multi_cpu_stop: curstate = 3, ack = 5
[ 1731.632619] migratio-14      1d..1 1729613680us : multi_cpu_stop: curstate = 3, ack = 5
[ 1731.632619] migratio-52      7d..1 1729613680us : multi_cpu_stop: curstate = 3, ack = 4
[ 1731.632619] migratio-11      0d..1 1729613680us : multi_cpu_stop: curstate = 3, ack = 4
[ 1731.632619] migratio-46      6d..1 1729613680us : sched_ttwu_pending: entered
[ 1731.632619] migratio-14      1d..1 1729613680us : multi_cpu_stop: curstate = 4, ack = 5
[ 1731.632619] migratio-46      6d..1 1729613680us : multi_cpu_stop: curstate = 4, ack = 5
[ 1731.632619] migratio-52      7d..1 1729613680us : multi_cpu_stop: curstate = 4, ack = 5
[ 1731.632619] migratio-21      2d..1 1729614637us : multi_cpu_stop: curstate = 4, ack = 5
[ 1731.632619] migratio-14      1d... 1729614817us : sched_switch: prev_comm=migration/1 prev_pid=14 prev_prio=0 prev_state=S ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] migratio-52      7d... 1729614820us : sched_switch: prev_comm=migration/7 prev_pid=52 prev_prio=0 prev_state=S ==> next_comm=rcu_torture_rea next_pid=129 next_prio=139
[ 1731.632619] rcu_tort-129     7d... 1729614828us : sched_switch: prev_comm=rcu_torture_rea prev_pid=129 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=128 next_prio=139
[ 1731.632619] rcu_tort-128     7d... 1729614833us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=S ==> next_comm=swapper/7 next_pid=0 next_prio=120
[ 1731.632619] rcu_tort-124     1d... 1729614834us : activate_task: entered
[ 1731.632619] rcu_tort-124     1d... 1729614836us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729614836us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614836us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614837us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614837us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614837us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614837us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729614838us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729614839us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_wri next_pid=119 next_prio=120
[ 1731.632619] migratio-11      0d..1 1729614839us : multi_cpu_stop: curstate = 4, ack = 1
[ 1731.632619] migratio-11      0d.h1 1729614974us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.h1 1729614974us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.h1 1729614976us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.h1 1729614977us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.h1 1729614977us : activate_task: entered
[ 1731.632619] migratio-11      0d.h1 1729614978us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.h1 1729614979us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0..s1 1729614988us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729614989us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729614990us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729614991us : try_to_wake_up: ttwu_queue_remote entered, CPU 7
[ 1731.632619] migratio-11      0..s1 1729615008us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615008us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615009us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615010us : try_to_wake_up: ttwu_queue_remote entered, CPU 7
[ 1731.632619] migratio-11      0..s1 1729615010us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615010us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615011us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615011us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619] rcu_tort-119     1d... 1729615025us : scheduler_ipi: entered
[ 1731.632619] rcu_tort-119     1d.h. 1729615026us : sched_ttwu_pending: entered
[ 1731.632619] rcu_tort-119     1d.h. 1729615026us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] rcu_tort-119     1d.h. 1729615026us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-119     1d.h. 1729615027us : activate_task: entered
[ 1731.632619] rcu_tort-119     1d.h. 1729615028us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-119     1d.h. 1729615028us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0d.s1 1729615029us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615029us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615030us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615030us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615030us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615031us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615031us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0d.s1 1729615032us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615032us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0..s1 1729615033us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615033us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615034us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615034us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615034us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615035us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615035us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0..s1 1729615036us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615036us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615038us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615038us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615038us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615039us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615039us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0d.s1 1729615040us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615040us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615042us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615042us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0..s1 1729615043us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615043us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615044us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615044us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615044us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615045us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615045us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0..s1 1729615046us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615046us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615046us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615047us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615047us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615047us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615047us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0..s1 1729615048us : wake_up_process: entered
[ 1731.632619] migratio-11      0..s1 1729615049us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615050us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615050us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615051us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615051us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615051us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0d.s1 1729615057us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615057us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615057us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-11      0d.s1 1729615058us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-11      0d.s1 1729615058us : activate_task: entered
[ 1731.632619] migratio-11      0d.s1 1729615058us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-11      0d.s1 1729615059us : check_preempt_curr: entered
[ 1731.632619] migratio-11      0d.s1 1729615060us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615060us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615061us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615061us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615094us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615094us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615095us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615095us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615096us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615096us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d.s1 1729615097us : wake_up_process: entered
[ 1731.632619] migratio-11      0d.s1 1729615097us : try_to_wake_up: entered
[ 1731.632619] migratio-11      0d... 1729615103us : sched_switch: prev_comm=migration/0 prev_pid=11 prev_prio=0 prev_state=S ==> next_comm=rcu_sched next_pid=10 next_prio=120
[ 1731.632619] rcu_sche-10      0d... 1729615108us : sched_switch: prev_comm=rcu_sched prev_pid=10 prev_prio=120 prev_state=I ==> next_comm=init next_pid=1 next_prio=120
[ 1731.632619] rcu_tort-119     1d... 1729615678us : sched_switch: prev_comm=rcu_torture_wri prev_pid=119 prev_prio=120 prev_state=S ==> next_comm=rcu_torture_rea next_pid=125 next_prio=139
[ 1731.632619] rcu_tort-125     1d... 1729615683us : sched_switch: prev_comm=rcu_torture_rea prev_pid=125 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619]   <idle>-0       7d... 1729615886us : scheduler_ipi: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616090us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616090us : sched_ttwu_pending: non-NULL llist
[ 1731.632619]   <idle>-0       7d.h. 1729616091us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616091us : activate_task: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616092us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616093us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       7d.h. 1729616093us : resched_curr: entered
[ 1731.632619]   <idle>-0       7dNh. 1729616094us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       7dNh. 1729616094us : activate_task: entered
[ 1731.632619]   <idle>-0       7dNh. 1729616095us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       7dNh. 1729616095us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       7dNh. 1729616117us : resched_curr: entered
[ 1731.632619]   <idle>-0       7.N.. 1729616153us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       7d... 1729616155us : sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=rcu_torture_rea next_pid=129 next_prio=139
[ 1731.632619] rcu_tort-129     7d... 1729616159us : sched_switch: prev_comm=rcu_torture_rea prev_pid=129 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=128 next_prio=139
[ 1731.632619]     init-1       0d... 1729616201us : sched_switch: prev_comm=init prev_pid=1 prev_prio=120 prev_state=S ==> next_comm=kworker/u16:5 next_pid=162 next_prio=120
[ 1731.632619] kworker/-162     0d... 1729616207us : wake_up_process: entered
[ 1731.632619] kworker/-162     0d... 1729616207us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0d... 1729616208us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] kworker/-162     0d... 1729616209us : ttwu_do_activate.isra.108: entered
[ 1731.632619] kworker/-162     0d... 1729616209us : activate_task: entered
[ 1731.632619] kworker/-162     0d... 1729616211us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] kworker/-162     0d... 1729616211us : check_preempt_curr: entered
[ 1731.632619] kworker/-162     0d... 1729616213us : wake_up_process: entered
[ 1731.632619] kworker/-162     0d... 1729616214us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0d... 1729616215us : wake_up_process: entered
[ 1731.632619] kworker/-162     0d... 1729616215us : try_to_wake_up: entered
[ 1731.632619] rcu_tort-124     1d.H. 1729618001us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729618015us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=127 next_prio=139
[ 1731.632619] rcu_tort-127     1d... 1729618022us : sched_switch: prev_comm=rcu_torture_rea prev_pid=127 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d.h. 1729620527us : resched_curr: entered
[ 1731.632619] rcu_tort-126     1d... 1729620533us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] rcu_tort-124     1d.h. 1729623526us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729623532us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d.h. 1729626526us : resched_curr: entered
[ 1731.632619] rcu_tort-126     1d... 1729626534us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] rcu_tort-124     1d.h. 1729629526us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729629719us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d.h. 1729632524us : resched_curr: entered
[ 1731.632619] rcu_tort-126     1d... 1729632677us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] rcu_tort-124     1d.h. 1729635526us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729635533us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d.h. 1729638531us : resched_curr: entered
[ 1731.632619] rcu_tort-126     1d... 1729638541us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] rcu_tort-124     1d.h. 1729641526us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729641534us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d.h. 1729644526us : resched_curr: entered
[ 1731.632619] rcu_tort-126     1d... 1729644622us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] rcu_tort-124     1d.h. 1729647528us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729647534us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=R+ ==> next_comm=rcu_torture_rea next_pid=126 next_prio=139
[ 1731.632619] rcu_tort-126     1d... 1729647662us : sched_switch: prev_comm=rcu_torture_rea prev_pid=126 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=124 next_prio=139
[ 1731.632619] kworker/-162     0d.h. 1729647665us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729647674us : activate_task: entered
[ 1731.632619] rcu_tort-124     1d... 1729647675us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729647675us : activate_task: entered
[ 1731.632619] rcu_tort-124     1d... 1729647675us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729647676us : resched_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647676us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647676us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647676us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647677us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647677us : activate_task: entered
[ 1731.632619] rcu_tort-124     1dN.. 1729647677us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-124     1d... 1729647679us : sched_switch: prev_comm=rcu_torture_rea prev_pid=124 prev_prio=139 prev_state=S ==> next_comm=torture_stutter next_pid=133 next_prio=120
[ 1731.632619] torture_-133     1d... 1729647682us : sched_switch: prev_comm=torture_stutter prev_pid=133 prev_prio=120 prev_state=S ==> next_comm=rcu_torture_fwd next_pid=136 next_prio=139
[ 1731.632619] kworker/-162     0.Ns. 1729647684us : wake_up_process: entered
[ 1731.632619] kworker/-162     0.Ns. 1729647685us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0dNs. 1729647688us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] kworker/-162     0dNs. 1729647690us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619] rcu_tort-128     7d... 1729647690us : sched_switch: prev_comm=rcu_torture_rea prev_pid=128 prev_prio=139 prev_state=S ==> next_comm=swapper/7 next_pid=0 next_prio=120
[ 1731.632619] kworker/-162     0.Ns. 1729647707us : wake_up_process: entered
[ 1731.632619] kworker/-162     0.Ns. 1729647707us : try_to_wake_up: entered
[ 1731.632619] rcu_tort-136     1d... 1729647709us : scheduler_ipi: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647709us : sched_ttwu_pending: entered
[ 1731.632619] kworker/-162     0dNs. 1729647709us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647709us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] rcu_tort-136     1d.h. 1729647710us : ttwu_do_activate.isra.108: entered
[ 1731.632619] kworker/-162     0dNs. 1729647710us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619] rcu_tort-136     1d.h. 1729647710us : activate_task: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647711us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647711us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-136     1.... 1729647714us : wake_up_process: entered
[ 1731.632619] rcu_tort-136     1.... 1729647714us : try_to_wake_up: entered
[ 1731.632619] rcu_tort-136     1d... 1729647716us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] rcu_tort-136     1d... 1729647716us : try_to_wake_up: ttwu_queue_remote entered, CPU 2
[ 1731.632619] kworker/-162     0.Ns. 1729647723us : wake_up_process: entered
[ 1731.632619] kworker/-162     0.Ns. 1729647723us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0dNs. 1729647724us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] kworker/-162     0dNs. 1729647724us : ttwu_do_activate.isra.108: entered
[ 1731.632619] kworker/-162     0dNs. 1729647725us : activate_task: entered
[ 1731.632619] kworker/-162     0dNs. 1729647726us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] kworker/-162     0dNs. 1729647726us : check_preempt_curr: entered
[ 1731.632619] kworker/-162     0.Ns. 1729647727us : wake_up_process: entered
[ 1731.632619] kworker/-162     0.Ns. 1729647727us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0dNs. 1729647728us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] kworker/-162     0dNs. 1729647729us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619] rcu_tort-136     1d... 1729647730us : scheduler_ipi: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647730us : sched_ttwu_pending: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647730us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] rcu_tort-136     1d.h. 1729647731us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647731us : activate_task: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647731us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647732us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-136     1d.h. 1729647732us : resched_curr: entered
[ 1731.632619] rcu_tort-136     1dNh. 1729647732us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-136     1dNh. 1729647732us : activate_task: entered
[ 1731.632619] rcu_tort-136     1dNh. 1729647733us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] kworker/-162     0d... 1729647733us : sched_switch: prev_comm=kworker/u16:5 prev_pid=162 prev_prio=120 prev_state=R+ ==> next_comm=kworker/0:3 next_pid=171 next_prio=120
[ 1731.632619] rcu_tort-136     1dNh. 1729647733us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-136     1d... 1729647734us : sched_switch: prev_comm=rcu_torture_fwd prev_pid=136 prev_prio=139 prev_state=D ==> next_comm=rcu_torture_wri next_pid=119 next_prio=120
[ 1731.632619] kworker/-171     0d... 1729647752us : sched_switch: prev_comm=kworker/0:3 prev_pid=171 prev_prio=120 prev_state=I ==> next_comm=kworker/u16:4 next_pid=161 next_prio=120
[ 1731.632619] kworker/-161     0d... 1729647754us : sched_switch: prev_comm=kworker/u16:4 prev_pid=161 prev_prio=120 prev_state=I ==> next_comm=rcu_sched next_pid=10 next_prio=120
[ 1731.632619] rcu_tort-119     1d... 1729648207us : sched_switch: prev_comm=rcu_torture_wri prev_pid=119 prev_prio=120 prev_state=D ==> next_comm=rcu_torture_sta next_pid=131 next_prio=120
[ 1731.632619] rcu_sche-10      0d... 1729648662us : resched_curr: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650719us : wake_up_process: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650719us : try_to_wake_up: entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650720us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-21      2dNs1 1729650720us : scheduler_ipi: entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650721us : try_to_wake_up: ttwu_queue_remote entered, CPU 7
[ 1731.632619] migratio-21      2dNH1 1729650721us : sched_ttwu_pending: entered
[ 1731.632619] migratio-21      2dNH1 1729650721us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] migratio-21      2dNH1 1729650721us : ttwu_do_activate.isra.108: entered
[ 1731.632619] migratio-21      2dNH1 1729650722us : activate_task: entered
[ 1731.632619] migratio-21      2dNH1 1729650723us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] migratio-21      2dNH1 1729650723us : check_preempt_curr: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650734us : wake_up_process: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650735us : try_to_wake_up: entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650735us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650736us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619]   <idle>-0       7d... 1729650736us : scheduler_ipi: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650747us : wake_up_process: entered
[ 1731.632619] rcu_sche-10      0..s. 1729650747us : try_to_wake_up: entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650748us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] rcu_sche-10      0d.s. 1729650748us : try_to_wake_up: ttwu_queue_remote entered, CPU 1
[ 1731.632619]   <idle>-0       7d.h. 1729650749us : sched_ttwu_pending: entered
[ 1731.632619]   <idle>-0       7d.h. 1729650749us : sched_ttwu_pending: non-NULL llist
[ 1731.632619]   <idle>-0       7d.h. 1729650750us : ttwu_do_activate.isra.108: entered
[ 1731.632619]   <idle>-0       7d.h. 1729650750us : activate_task: entered
[ 1731.632619] rcu_sche-10      0d... 1729650754us : sched_switch: prev_comm=rcu_sched prev_pid=10 prev_prio=120 prev_state=I ==> next_comm=kworker/u16:5 next_pid=162 next_prio=120
[ 1731.632619]   <idle>-0       7d.h. 1729650755us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619]   <idle>-0       7d.h. 1729650755us : check_preempt_curr: entered
[ 1731.632619]   <idle>-0       7d.h. 1729650755us : resched_curr: entered
[ 1731.632619] migratio-21      2dNs1 1729650758us : wake_up_process: entered
[ 1731.632619] migratio-21      2dNs1 1729650758us : try_to_wake_up: entered
[ 1731.632619] migratio-21      2dNs1 1729650759us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-21      2dNs1 1729650759us : try_to_wake_up: ttwu_queue_remote entered, CPU 0
[ 1731.632619] kworker/-162     0d... 1729650763us : activate_task: entered
[ 1731.632619] kworker/-162     0d... 1729650764us : check_preempt_curr: entered
[ 1731.632619] kworker/-162     0d... 1729650765us : activate_task: entered
[ 1731.632619] kworker/-162     0d... 1729650765us : check_preempt_curr: entered
[ 1731.632619] kworker/-162     0d... 1729650765us : activate_task: entered
[ 1731.632619] kworker/-162     0d... 1729650766us : check_preempt_curr: entered
[ 1731.632619] migratio-21      2dN.1 1729650770us : try_to_wake_up: entered
[ 1731.632619] kworker/-162     0d... 1729650770us : sched_switch: prev_comm=kworker/u16:5 prev_pid=162 prev_prio=120 prev_state=I ==> next_comm=rcu_torture_rea next_pid=125 next_prio=139
[ 1731.632619] migratio-21      2dN.1 1729650771us : try_to_wake_up: ttwu_queue entered
[ 1731.632619] migratio-21      2dN.1 1729650771us : try_to_wake_up: ttwu_queue_remote entered, CPU 0
[ 1731.632619] migratio-21      2d... 1729650774us : sched_switch: prev_comm=migration/2 prev_pid=21 prev_prio=0 prev_state=S ==> next_comm=rcuog/1 next_pid=18 next_prio=120
[ 1731.632619] rcu_tort-125     0d... 1729650775us : scheduler_ipi: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650775us : sched_ttwu_pending: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650775us : sched_ttwu_pending: non-NULL llist
[ 1731.632619] rcu_tort-125     0d.h. 1729650776us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650777us : activate_task: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650778us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650778us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650779us : ttwu_do_activate.isra.108: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650779us : activate_task: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650779us : ttwu_do_wakeup.isra.107: entered
[ 1731.632619] rcu_tort-125     0d.h. 1729650779us : check_preempt_curr: entered
[ 1731.632619] rcu_tort-125     0d... 1729650783us : sched_switch: prev_comm=rcu_torture_rea prev_pid=125 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=127 next_prio=139
[ 1731.632619]  rcuog/1-18      2d... 1729650783us : sched_switch: prev_comm=rcuog/1 prev_pid=18 prev_prio=120 prev_state=S ==> next_comm=swapper/2 next_pid=0 next_prio=120
[ 1731.632619] rcu_tort-127     0d... 1729650786us : sched_switch: prev_comm=rcu_torture_rea prev_pid=127 prev_prio=139 prev_state=S ==> next_comm=rcu_torture_rea next_pid=130 next_prio=139
[ 1731.632619] rcu_tort-130     0d... 1729650789us : sched_switch: prev_comm=rcu_torture_rea prev_pid=130 prev_prio=139 prev_state=S ==> next_comm=torture_onoff next_pid=135 next_prio=120
[ 1731.632619] ---------------------------------
[ 1795.802039] smpboot: CPU 6 is now offline
#40 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 58 lines · Paul E. McKenney
Quoted 42 lines · Paul E. McKenney
Quoted 27 lines · Paul E. McKenney
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 7 lines
Quoted 2 lines

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

And I did eventually get a good trace. If I am interpreting this trace correctly, the torture_-135 task didn't get around to attempting to wake up all of the CPUs. I will try again, but this time with the sched_switch trace event enabled.

Quoted 7 lines

As a side note, enabling ftrace from the command line seems to interact badly with turning tracing off and on in the kernel, so I eventually resorted to trace_printk() in the functions of interest. The trace output is below, followed by the current diagnostic patch. Please note that I am -not- using the desperation hammer-the-scheduler patches.

More as I learn more!

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

Quoted 7 lines

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Functionally, multi_cpu_stop() is working fine, according to the trace below (search for a line beginning with TAB). But somehow CPU 2 took almost three -minutes- to do one iteration of the loop. The prime suspect in that loop is cpu_relax() due to the hypervisor having an opportunity to do something at that point. The commentary below (again, search for a line beginning with TAB) gives my analysis.

Quoted 8 lines

Of course, if I am correct, it should be possible to catch cpu_relax() in the act. That is the next step, give or take the Heisenbuggy nature of this beast.

Another thing for me to try is to run longer with !NO_HZ_FULL, just in case the earlier runs just got lucky.

Thoughts?

And it really can happen:

[ 1881.467922] migratio-33      4...1 1879530317us : stop_machine_yield: cpu_relax() took 756140 ms

The previous timestamp was 1123391100us, so the cpu_relax() is almost exactly the full delay.

But another instance stalled for many minutes without a ten-second cpu_relax(). So it is not just cpu_relax() causing trouble. I could rationalize that vCPU preemption being at fault...

And my diagnostic patch is below, just in case I am doing something stupid with that.

Thanx, Paul

------------------------------------------------------------------------

kernel/rcu/tree.c +1 -0
index ce00b442ced0..1a50ed258ef0 100644
@@ -3569,6 +3569,7 @@ void __init rcu_init(void)
 	rcu_par_gp_wq = alloc_workqueue("rcu_par_gp", WQ_MEM_RECLAIM, 0);
 	WARN_ON(!rcu_par_gp_wq);
 	srcu_init();
+	tracing_off();
 }
 
 #include "tree_stall.h"
kernel/sched/core.c +13 -0
index 0b22e55cebe8..a5a879a49051 100644
@@ -479,6 +479,7 @@ void wake_up_q(struct wake_q_head *head)
 {
 	struct wake_q_node *node = head->first;
 
+	trace_printk("entered\n");
 	while (node != WAKE_Q_TAIL) {
 		struct task_struct *task;
 
@@ -509,6 +510,7 @@ void resched_curr(struct rq *rq)
 	struct task_struct *curr = rq->curr;
 	int cpu;
 
+	trace_printk("entered\n");
 	lockdep_assert_held(&rq->lock);
 
 	if (test_tsk_need_resched(curr))
@@ -1197,6 +1199,7 @@ static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
 
 void activate_task(struct rq *rq, struct task_struct *p, int flags)
 {
+	trace_printk("entered\n");
 	if (task_contributes_to_load(p))
 		rq->nr_uninterruptible--;
 
@@ -1298,6 +1301,7 @@ void check_preempt_curr(struct rq *rq, struct task_struct *p, int flags)
 {
 	const struct sched_class *class;
 
+	trace_printk("entered\n");
 	if (p->sched_class == rq->curr->sched_class) {
 		rq->curr->sched_class->check_preempt_curr(rq, p, flags);
 	} else {
@@ -2097,6 +2101,7 @@ ttwu_stat(struct task_struct *p, int cpu, int wake_flags)
 static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
 			   struct rq_flags *rf)
 {
+	trace_printk("entered\n");
 	check_preempt_curr(rq, p, wake_flags);
 	p->state = TASK_RUNNING;
 	trace_sched_wakeup(p);
@@ -2132,6 +2137,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 {
 	int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
 
+	trace_printk("entered\n");
 	lockdep_assert_held(&rq->lock);
 
 #ifdef CONFIG_SMP
@@ -2178,9 +2184,11 @@ void sched_ttwu_pending(void)
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
+	trace_printk("entered\n");
 	if (!llist)
 		return;
 
+	trace_printk("non-NULL llist\n");
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
@@ -2192,6 +2200,7 @@ void sched_ttwu_pending(void)
 
 void scheduler_ipi(void)
 {
+	trace_printk("entered\n");
 	/*
 	 * Fold TIF_NEED_RESCHED into the preempt_count; anybody setting
 	 * TIF_NEED_RESCHED remotely (for the first time) will also send
@@ -2232,6 +2241,7 @@ static void ttwu_queue_remote(struct task_struct *p, int cpu, int wake_flags)
 {
 	struct rq *rq = cpu_rq(cpu);
 
+	trace_printk("%s entered, CPU %d\n", __func__, cpu);
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	if (llist_add(&p->wake_entry, &cpu_rq(cpu)->wake_list)) {
@@ -2277,6 +2287,7 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
 	struct rq *rq = cpu_rq(cpu);
 	struct rq_flags rf;
 
+	trace_printk("%s entered\n", __func__);
 #if defined(CONFIG_SMP)
 	if (sched_feat(TTWU_QUEUE) && !cpus_share_cache(smp_processor_id(), cpu)) {
 		sched_clock_cpu(cpu); /* Sync clocks across CPUs */
@@ -2399,6 +2410,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 	unsigned long flags;
 	int cpu, success = 0;
 
+	trace_printk("entered\n");
 	preempt_disable();
 	if (p == current) {
 		/*
@@ -2545,6 +2557,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
  */
 int wake_up_process(struct task_struct *p)
 {
+	trace_printk("entered\n");
 	return try_to_wake_up(p, TASK_NORMAL, 0);
 }
 EXPORT_SYMBOL(wake_up_process);
kernel/stop_machine.c +24 -3
index 5c2b2f90fae1..a07f77b9c1f2 100644
@@ -21,6 +21,7 @@
 #include <linux/atomic.h>
 #include <linux/nmi.h>
 #include <linux/sched/wake_q.h>
+#include <linux/sched/clock.h>
 
 /*
  * Structure to determine completion condition and record errors.  May
@@ -80,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work)
 	unsigned long flags;
 	bool enabled;
 
+	trace_printk("entered for CPU %u\n", cpu);
 	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
@@ -167,7 +169,7 @@ static void set_state(struct multi_stop_data *msdata,
 	/* Reset ack counter. */
 	atomic_set(&msdata->thread_ack, msdata->num_threads);
 	smp_wmb();
-	msdata->state = newstate;
+	WRITE_ONCE(msdata->state, newstate);
 }
 
 /* Last one to ack a state moves to the next state. */
@@ -179,7 +181,15 @@ static void ack_state(struct multi_stop_data *msdata)
 
 void __weak stop_machine_yield(const struct cpumask *cpumask)
 {
+	u64 starttime = local_clock();
+	u64 endtime;
+	const u64 delta = 100ULL * 1000ULL * 1000ULL * 1000ULL;
+
 	cpu_relax();
+	endtime = local_clock();
+	if (time_after64(endtime, starttime + delta))
+		trace_printk("cpu_relax() took %llu ms\n",
+			     (endtime - starttime) / (1000ULL * 1000ULL));
 }
 
 /* This is the cpu_stop function which stops the CPU. */
@@ -210,8 +220,9 @@ static int multi_cpu_stop(void *data)
 	do {
 		/* Chill out and ensure we re-read multi_stop_state. */
 		stop_machine_yield(cpumask);
-		if (msdata->state != curstate) {
-			curstate = msdata->state;
+		if (READ_ONCE(msdata->state) != curstate) {
+			curstate = READ_ONCE(msdata->state);
+			trace_printk("curstate = %d, ack = %d\n", curstate, atomic_read(&msdata->thread_ack));
 			switch (curstate) {
 			case MULTI_STOP_DISABLE_IRQ:
 				local_irq_disable();
@@ -382,6 +393,7 @@ static bool queue_stop_cpus_work(const struct cpumask *cpumask,
 	 * preempted by a stopper which might wait for other stoppers
 	 * to enter @fn which can lead to deadlock.
 	 */
+	trace_printk("entered\n");
 	preempt_disable();
 	stop_cpus_in_progress = true;
 	for_each_cpu(cpu, cpumask) {
@@ -402,11 +414,18 @@ static int __stop_cpus(const struct cpumask *cpumask,
 		       cpu_stop_fn_t fn, void *arg)
 {
 	struct cpu_stop_done done;
+	unsigned long j = jiffies;
 
+	tracing_on();
+	trace_printk("entered\n");
+	trace_printk("CPUs %*pbl online\n", cpumask_pr_args(cpu_online_mask));
 	cpu_stop_init_done(&done, cpumask_weight(cpumask));
 	if (!queue_stop_cpus_work(cpumask, fn, arg, &done))
 		return -ENOENT;
 	wait_for_completion(&done.completion);
+	tracing_off();
+	if (time_after(jiffies, j + HZ * 20))
+		ftrace_dump(DUMP_ALL);
 	return done.ret;
 }
 
@@ -442,6 +461,7 @@ int stop_cpus(const struct cpumask *cpumask, cpu_stop_fn_t fn, void *arg)
 {
 	int ret;
 
+	trace_printk("entered\n");
 	/* static works are used, process one request at a time */
 	mutex_lock(&stop_cpus_mutex);
 	ret = __stop_cpus(cpumask, fn, arg);
@@ -599,6 +619,7 @@ int stop_machine_cpuslocked(cpu_stop_fn_t fn, void *data,
 		.active_cpus = cpus,
 	};
 
+	trace_printk("entered\n");
 	lockdep_assert_cpus_held();
 
 	if (!stop_machine_initialized) {
#41 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 75 lines · Paul E. McKenney
Quoted 58 lines · Paul E. McKenney
Quoted 42 lines · Paul E. McKenney
Quoted 27 lines · Paul E. McKenney
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Paul E. McKenney
Quoted 2 lines

Right; so clearly we're not understanding what's happening. That seems like a requirement for actually doing a patch.

Almost but not quite. It is a requirement for a patch *that* *is* *supposed* *to* *be* *a* *fix*. If you are trying to prohibit me from writing experimental patches, please feel free to take a long walk on a short pier.

Understood???

Quoted 7 lines
Quoted 2 lines

Ah, my bad, I thought you were actually proposing this as an actual patch. I now see that is my bad, I'd overlooked the RFC part.

No problem!

And of course adding tracing decreases the frequency and duration of the multi_cpu_stop(). Re-running with shorter-duration triggering. ;-)

And I did eventually get a good trace. If I am interpreting this trace correctly, the torture_-135 task didn't get around to attempting to wake up all of the CPUs. I will try again, but this time with the sched_switch trace event enabled.

Quoted 7 lines

As a side note, enabling ftrace from the command line seems to interact badly with turning tracing off and on in the kernel, so I eventually resorted to trace_printk() in the functions of interest. The trace output is below, followed by the current diagnostic patch. Please note that I am -not- using the desperation hammer-the-scheduler patches.

More as I learn more!

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

Quoted 7 lines

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Functionally, multi_cpu_stop() is working fine, according to the trace below (search for a line beginning with TAB). But somehow CPU 2 took almost three -minutes- to do one iteration of the loop. The prime suspect in that loop is cpu_relax() due to the hypervisor having an opportunity to do something at that point. The commentary below (again, search for a line beginning with TAB) gives my analysis.

Quoted 8 lines

Of course, if I am correct, it should be possible to catch cpu_relax() in the act. That is the next step, give or take the Heisenbuggy nature of this beast.

Another thing for me to try is to run longer with !NO_HZ_FULL, just in case the earlier runs just got lucky.

Thoughts?

And it really can happen:

[ 1881.467922] migratio-33      4...1 1879530317us : stop_machine_yield: cpu_relax() took 756140 ms

The previous timestamp was 1123391100us, so the cpu_relax() is almost exactly the full delay.

Quoted 6 lines

But another instance stalled for many minutes without a ten-second cpu_relax(). So it is not just cpu_relax() causing trouble. I could rationalize that vCPU preemption being at fault...

And my diagnostic patch is below, just in case I am doing something stupid with that.

I did a 12-hour run with the same configuration except for leaving out the "nohz_full=1-7" kernel parameter without problems (aside from the RCU CPU stall warnings due to the ftrace_dump() at the end of the run -- isn't there some way to clear the ftrace buffer without actually printing it?).

My next step is to do an over-the-weekend run with the same configuration, then a similar run with more recent kernel and qemu but with the "nohz_full=1-7". If both of those pass, I will consider this to be a KVM/qemu bug that has since been fixed.

Thanx, Paul

#42 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 41 lines · Paul E. McKenney
Quoted 41 lines
Quoted 21 lines
Quoted 6 lines

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

Quoted 7 lines

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Functionally, multi_cpu_stop() is working fine, according to the trace below (search for a line beginning with TAB). But somehow CPU 2 took almost three -minutes- to do one iteration of the loop. The prime suspect in that loop is cpu_relax() due to the hypervisor having an opportunity to do something at that point. The commentary below (again, search for a line beginning with TAB) gives my analysis.

Quoted 8 lines

Of course, if I am correct, it should be possible to catch cpu_relax() in the act. That is the next step, give or take the Heisenbuggy nature of this beast.

Another thing for me to try is to run longer with !NO_HZ_FULL, just in case the earlier runs just got lucky.

Thoughts?

And it really can happen:

[ 1881.467922] migratio-33      4...1 1879530317us : stop_machine_yield: cpu_relax() took 756140 ms

The previous timestamp was 1123391100us, so the cpu_relax() is almost exactly the full delay.

But another instance stalled for many minutes without a ten-second cpu_relax(). So it is not just cpu_relax() causing trouble. I could rationalize that vCPU preemption being at fault...

Quoted 7 lines
Quoted 2 lines

And my diagnostic patch is below, just in case I am doing something stupid with that.

I did a 12-hour run with the same configuration except for leaving out the "nohz_full=1-7" kernel parameter without problems (aside from the RCU CPU stall warnings due to the ftrace_dump() at the end of the run -- isn't there some way to clear the ftrace buffer without actually printing it?).

I think if tracing_reset_all_online_cpus() is exposed, then calling that with the appropriate locks held can reset the ftrace ring buffer and clear the trace. Steve, is there a better way?

thanks,

- Joel

#43 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 45 lines · Joel Fernandes
Quoted 44 lines · Paul E. McKenney
Quoted 43 lines
Quoted 21 lines
Quoted 6 lines

And of course I forgot to dump out the online CPUs, so I really had no idea whether or not all the CPUs were accounted for. I added tracing to dump out the online CPUs at the beginning of __stop_cpus() and then reworked it a few times to get the problem to happen in reasonable time. Please see below for the resulting annotated trace.

Quoted 7 lines

I was primed to expect a lost IPI, perhaps due to yet another qemu bug, but all the migration threads are running within about 2 milliseconds. It is then almost two minutes(!) until the next trace message.

Looks like time to (very carefully!) instrument multi_cpu_stop().

Of course, if you have any ideas, please do not keep them a secret!

Functionally, multi_cpu_stop() is working fine, according to the trace below (search for a line beginning with TAB). But somehow CPU 2 took almost three -minutes- to do one iteration of the loop. The prime suspect in that loop is cpu_relax() due to the hypervisor having an opportunity to do something at that point. The commentary below (again, search for a line beginning with TAB) gives my analysis.

Quoted 8 lines

Of course, if I am correct, it should be possible to catch cpu_relax() in the act. That is the next step, give or take the Heisenbuggy nature of this beast.

Another thing for me to try is to run longer with !NO_HZ_FULL, just in case the earlier runs just got lucky.

Thoughts?

And it really can happen:

[ 1881.467922] migratio-33      4...1 1879530317us : stop_machine_yield: cpu_relax() took 756140 ms

The previous timestamp was 1123391100us, so the cpu_relax() is almost exactly the full delay.

But another instance stalled for many minutes without a ten-second cpu_relax(). So it is not just cpu_relax() causing trouble. I could rationalize that vCPU preemption being at fault...

And my diagnostic patch is below, just in case I am doing something stupid with that.

Quoted 8 lines
Quoted 4 lines

I did a 12-hour run with the same configuration except for leaving out the "nohz_full=1-7" kernel parameter without problems (aside from the RCU CPU stall warnings due to the ftrace_dump() at the end of the run -- isn't there some way to clear the ftrace buffer without actually printing it?).

I think if tracing_reset_all_online_cpus() is exposed, then calling that with the appropriate locks held can reset the ftrace ring buffer and clear the trace. Steve, is there a better way?

On the off-chance that it helps, here is my use case:

o I have a race condition that becomes extremely unlikely if
I leave tracing enabled at all times.

o I therefore enable tracing at the beginning of a CPU-hotplug
removal.

o At the end of that CPU-hotplug removal, I disable tracing.

o I can test whether the problem occurred without affecting its
probability. If it occurred, I want to dump only that portion of the trace buffer since enabling it above. If the problem did not occur, I want to flush (not dump) the trace buffer.

o I cannot reasonably just make the trace buffer small because
the number of events in a given occurrence of the problem can vary widely.

Thanx, Paul

#44 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 27 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

---
 kernel/stop_machine.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/stop_machine.c +1 -0
index b4f83f7bdf86..a2659f61ed92 100644
@@ -20,6 +20,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/tick.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
 {

kernel/stop_machine.c static int multi_cpu_stop(void *data) · line 197

Quoted 10 lines
kernel/stop_machine.c +3 -1
 	struct multi_stop_data *msdata = data;
 	enum multi_stop_state curstate = MULTI_STOP_NONE;
-	int cpu = smp_processor_id(), err = 0;
+	int cpu, err = 0;
 	const struct cpumask *cpumask;
 	unsigned long flags;
 	bool is_active;
 
+	for_each_online_cpu(cpu)
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);

Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Thanks.

#45 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 36 lines · Frederic Weisbecker
Quoted 35 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

---
 kernel/stop_machine.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/stop_machine.c +2 -1
index b4f83f7bdf86..a2659f61ed92 100644
@@ -20,6 +20,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/tick.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
 {
 	struct multi_stop_data *msdata = data;
 	enum multi_stop_state curstate = MULTI_STOP_NONE;
-	int cpu = smp_processor_id(), err = 0;
+	int cpu, err = 0;
 	const struct cpumask *cpumask;
 	unsigned long flags;
 	bool is_active;
 
Quoted 5 lines

kernel/stop_machine.c static int multi_cpu_stop(void *data) · line 197

Quoted 2 lines
kernel/stop_machine.c +2 -0
+	for_each_online_cpu(cpu)
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);

Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(), which would be simple wrappers around RCU's private rcu_dynticks_snap() and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner case or three?

Thanx, Paul

#46 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 59 lines · Paul E. McKenney
Quoted 36 lines · Frederic Weisbecker
Quoted 35 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

---
 kernel/stop_machine.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/stop_machine.c +2 -1
index b4f83f7bdf86..a2659f61ed92 100644
@@ -20,6 +20,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/tick.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
 {
    struct multi_stop_data *msdata = data;
    enum multi_stop_state curstate = MULTI_STOP_NONE;
-   int cpu = smp_processor_id(), err = 0;
+   int cpu, err = 0;
    const struct cpumask *cpumask;
    unsigned long flags;
    bool is_active;
 
Quoted 5 lines

kernel/stop_machine.c static int multi_cpu_stop(void *data) · line 197

Quoted 2 lines
kernel/stop_machine.c +2 -0
+   for_each_online_cpu(cpu)
+           tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);

Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

Quoted 8 lines
        int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

        ...

        if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
                nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
        else
                /* Turn on the tick! */

This solution does make sense to me and is simpler than the rcu_nmi_exit_common() solution you proposed in the other thread.

Though I am not sure about the intricacies of remotely enabling the timer tick for a CPU. But overall, the above solution does seem simpler.

thanks,

- Joel

#47 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 29 lines · Paul E. McKenney
Quoted 3 lines · Frederic Weisbecker

On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote:
Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */
Quoted 6 lines

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(), which would be simple wrappers around RCU's private rcu_dynticks_snap() and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner case or three?

Oh I see. Until now we considered than running into the kernel (between user/guest/idle) is supposed to be short but there can be specific places where it doesn't apply.

I'm wondering if, more than just providing wrappers, this shouldn't be entirely driven by RCU using the tick_set_dep_cpu()/tick_clear_dep_cpu() at appropriate timings.

I don't want to sound like I'm trying to put all the work on you :p It's just that the tick shouldn't know much about RCU, it's rather RCU that is a client for the tick and is probably better suited to determine when a CPU becomes annoying with its extended grace period.

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

#48 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 43 lines · Frederic Weisbecker
Quoted 29 lines · Paul E. McKenney
Quoted 3 lines · Frederic Weisbecker

On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote:
Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */
Quoted 6 lines

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(), which would be simple wrappers around RCU's private rcu_dynticks_snap() and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner case or three?

Oh I see. Until now we considered than running into the kernel (between user/guest/idle) is supposed to be short but there can be specific places where it doesn't apply.

I'm wondering if, more than just providing wrappers, this shouldn't be entirely driven by RCU using the tick_set_dep_cpu()/tick_clear_dep_cpu() at appropriate timings.

Quoted 8 lines

I don't want to sound like I'm trying to put all the work on you :p It's just that the tick shouldn't know much about RCU, it's rather RCU that is a client for the tick and is probably better suited to determine when a CPU becomes annoying with its extended grace period.

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

Left to itself, RCU would take action only when a given nohz_full in-kernel CPU was delaying a grace period, which is what the (lightly tested) patch below is supposed to help with. If that is all that is needed, well and good!

But should we need long-running in-kernel nohz_full CPUs to turn on their ticks when they are not blocking an RCU grace period, for example, when RCU is idle, more will be needed. To that point, isn't there some sort of monitoring that checks up on nohz_full CPUs ever second or so? If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Thanx, Paul

------------------------------------------------------------------------

commit 1cb89508804f6f2fdb79a1be032b1932d52318c4

Author: Paul E. McKenney <[email protected]>
Date:   Mon Aug 12 16:14:00 2019 -0700

rcu: Force tick on for nohz_full CPUs not reaching quiescent states

CPUs running for long time periods in the kernel in nohz_full mode might leave the scheduling-clock interrupt disabled for then full duration of their in-kernel execution. This can (among other things) delay grace periods. This commit therefore forces the tick back on for any nohz_full CPU that is failing to pass through a quiescent state upon return from interrupt, which the resched_cpu() will induce.

kernel/rcu/tree.c +29 -7
index 8c494a692728..8b8f5bffdc5a 100644
@@ -651,6 +651,12 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	 */
 	if (rdp->dynticks_nmi_nesting != 1) {
 		trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2, rdp->dynticks);
+		if (tick_nohz_full_cpu(rdp->cpu) &&
+		    rdp->dynticks_nmi_nesting == 2 &&
+		    rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+			rdp->rcu_forced_tick = true;
+			tick_dep_set_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+		}
 		WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
 			   rdp->dynticks_nmi_nesting - 2);
 		return;
@@ -886,6 +892,16 @@ void rcu_irq_enter_irqson(void)
 	local_irq_restore(flags);
 }
 
+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+		tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+}
+
 /**
  * rcu_is_watching - see if RCU thinks that the current CPU is not idle
  *
@@ -1980,6 +1996,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 		if (!offloaded)
 			needwake = rcu_accelerate_cbs(rnp, rdp);
 
+		rcu_disable_tick_upon_qs(rdp);
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
 		/* ^^^ Released rnp->lock */
 		if (needwake)
@@ -2269,6 +2286,7 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 	int cpu;
 	unsigned long flags;
 	unsigned long mask;
+	struct rcu_data *rdp;
 	struct rcu_node *rnp;
 
 	rcu_for_each_leaf_node(rnp) {
@@ -2293,8 +2311,11 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 		for_each_leaf_node_possible_cpu(rnp, cpu) {
 			unsigned long bit = leaf_node_cpu_bit(rnp, cpu);
 			if ((rnp->qsmask & bit) != 0) {
-				if (f(per_cpu_ptr(&rcu_data, cpu)))
+				rdp = per_cpu_ptr(&rcu_data, cpu);
+				if (f(rdp)) {
 					mask |= bit;
+					rcu_disable_tick_upon_qs(rdp);
+				}
 			}
 		}
 		if (mask != 0) {
@@ -2322,7 +2343,7 @@ void rcu_force_quiescent_state(void)
 	rnp = __this_cpu_read(rcu_data.mynode);
 	for (; rnp != NULL; rnp = rnp->parent) {
 		ret = (READ_ONCE(rcu_state.gp_flags) & RCU_GP_FLAG_FQS) ||
-		      !raw_spin_trylock(&rnp->fqslock);
+		       !raw_spin_trylock(&rnp->fqslock);
 		if (rnp_old != NULL)
 			raw_spin_unlock(&rnp_old->fqslock);
 		if (ret)
@@ -2855,7 +2876,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp)
 {
 	if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) {
 		rcu_barrier_trace(TPS("LastCB"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		complete(&rcu_state.barrier_completion);
 	} else {
 		rcu_barrier_trace(TPS("CB"), -1, rcu_state.barrier_sequence);
@@ -2879,7 +2900,7 @@ static void rcu_barrier_func(void *unused)
 	} else {
 		debug_rcu_head_unqueue(&rdp->barrier_head);
 		rcu_barrier_trace(TPS("IRQNQ"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 	}
 	rcu_nocb_unlock(rdp);
 }
@@ -2906,7 +2927,7 @@ void rcu_barrier(void)
 	/* Did someone else do our work for us? */
 	if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
 		rcu_barrier_trace(TPS("EarlyExit"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		smp_mb(); /* caller's subsequent code after above check. */
 		mutex_unlock(&rcu_state.barrier_mutex);
 		return;
@@ -2938,11 +2959,11 @@ void rcu_barrier(void)
 			continue;
 		if (rcu_segcblist_n_cbs(&rdp->cblist)) {
 			rcu_barrier_trace(TPS("OnlineQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 			smp_call_function_single(cpu, rcu_barrier_func, NULL, 1);
 		} else {
 			rcu_barrier_trace(TPS("OnlineNQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 		}
 	}
 	put_online_cpus();
@@ -3168,6 +3189,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
 	} else {
kernel/rcu/tree.h +1 -0
index c612f306fe89..055c31781d3a 100644
@@ -181,6 +181,7 @@ struct rcu_data {
 	atomic_t dynticks;		/* Even value for idle, else odd. */
 	bool rcu_need_heavy_qs;		/* GP old, so heavy quiescent state! */
 	bool rcu_urgent_qs;		/* GP old need light quiescent state. */
+	bool rcu_forced_tick;		/* Forced tick to provide QS. */
 #ifdef CONFIG_RCU_FAST_NO_HZ
 	bool all_lazy;			/* All CPU's CBs lazy at idle start? */
 	unsigned long last_accelerate;	/* Last jiffy CBs were accelerated. */
#56 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 37 lines · Paul E. McKenney
Quoted 36 lines · Frederic Weisbecker
Quoted 35 lines · Paul E. McKenney

The multi_cpu_stop() function relies on the scheduler to gain control from whatever is running on the various online CPUs, including any nohz_full CPUs running long loops in kernel-mode code. Lack of the scheduler-clock interrupt on such CPUs can delay multi_cpu_stop() for several minutes and can also result in RCU CPU stall warnings. This commit therefore causes multi_cpu_stop() to enable the scheduler-clock interrupt on all online CPUs.

---
 kernel/stop_machine.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/stop_machine.c +2 -1
index b4f83f7bdf86..a2659f61ed92 100644
@@ -20,6 +20,7 @@
 #include <linux/smpboot.h>
 #include <linux/atomic.h>
 #include <linux/nmi.h>
+#include <linux/tick.h>
 #include <linux/sched/wake_q.h>
 
 /*
@@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
 {
 	struct multi_stop_data *msdata = data;
 	enum multi_stop_state curstate = MULTI_STOP_NONE;
-	int cpu = smp_processor_id(), err = 0;
+	int cpu, err = 0;
 	const struct cpumask *cpumask;
 	unsigned long flags;
 	bool is_active;
 
Quoted 5 lines
Quoted 5 lines

kernel/stop_machine.c static int multi_cpu_stop(void *data) · line 197

Quoted 2 lines
kernel/stop_machine.c +2 -0
+	for_each_online_cpu(cpu)
+		tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);

Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Except that I am not finding anything resembling tick_set_dep() in current mainline. What should I be looking for instead?

Thanx, Paul

#49 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

In reply to Paul E. McKenney
Quoted 58 lines · Paul E. McKenney
Quoted 43 lines · Frederic Weisbecker
Quoted 29 lines · Paul E. McKenney
Quoted 3 lines · Frederic Weisbecker

On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote:
Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */
Quoted 6 lines

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(), which would be simple wrappers around RCU's private rcu_dynticks_snap() and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner case or three?

Oh I see. Until now we considered than running into the kernel (between user/guest/idle) is supposed to be short but there can be specific places where it doesn't apply.

I'm wondering if, more than just providing wrappers, this shouldn't be entirely driven by RCU using the tick_set_dep_cpu()/tick_clear_dep_cpu() at appropriate timings.

Quoted 8 lines

I don't want to sound like I'm trying to put all the work on you :p It's just that the tick shouldn't know much about RCU, it's rather RCU that is a client for the tick and is probably better suited to determine when a CPU becomes annoying with its extended grace period.

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

Left to itself, RCU would take action only when a given nohz_full in-kernel CPU was delaying a grace period, which is what the (lightly tested) patch below is supposed to help with. If that is all that is needed, well and good!

Quoted 4 lines

But should we need long-running in-kernel nohz_full CPUs to turn on their ticks when they are not blocking an RCU grace period, for example, when RCU is idle, more will be needed. To that point, isn't there some sort of monitoring that checks up on nohz_full CPUs ever second or so?

Wouldn't such monitoring need to be more often than a second, given that rcu_urgent_qs and rcu_need_heavy_qs are configured typically to be sooner (200-300 jiffies on my system).

Quoted 9 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

2 comments below:

Quoted 69 lines

------------------------------------------------------------------------

commit 1cb89508804f6f2fdb79a1be032b1932d52318c4

Author: Paul E. McKenney <[email protected]>
Date:   Mon Aug 12 16:14:00 2019 -0700

rcu: Force tick on for nohz_full CPUs not reaching quiescent states

CPUs running for long time periods in the kernel in nohz_full mode might leave the scheduling-clock interrupt disabled for then full duration of their in-kernel execution. This can (among other things) delay grace periods. This commit therefore forces the tick back on for any nohz_full CPU that is failing to pass through a quiescent state upon return from interrupt, which the resched_cpu() will induce.

kernel/rcu/tree.c +18 -0
index 8c494a692728..8b8f5bffdc5a 100644
@@ -651,6 +651,12 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	 */
 	if (rdp->dynticks_nmi_nesting != 1) {
 		trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2, rdp->dynticks);
+		if (tick_nohz_full_cpu(rdp->cpu) &&
+		    rdp->dynticks_nmi_nesting == 2 &&
+		    rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+			rdp->rcu_forced_tick = true;
+			tick_dep_set_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+		}
 		WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
 			   rdp->dynticks_nmi_nesting - 2);
 		return;
@@ -886,6 +892,16 @@ void rcu_irq_enter_irqson(void)
 	local_irq_restore(flags);
 }
 
+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+		tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+}
+
 /**
  * rcu_is_watching - see if RCU thinks that the current CPU is not idle
  *
@@ -1980,6 +1996,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 		if (!offloaded)
 			needwake = rcu_accelerate_cbs(rnp, rdp);
 
+		rcu_disable_tick_upon_qs(rdp);
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
 		/* ^^^ Released rnp->lock */
 		if (needwake)
@@ -2269,6 +2286,7 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 	int cpu;
 	unsigned long flags;
 	unsigned long mask;
+	struct rcu_data *rdp;
 	struct rcu_node *rnp;
 
 	rcu_for_each_leaf_node(rnp) {

kernel/rcu/tree.c static void force_qs_rnp(int (*f)(struct rcu_data *rdp)) · line 2318

Quoted 9 lines
kernel/rcu/tree.c +4 -1
@@ -2293,8 +2311,11 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 		for_each_leaf_node_possible_cpu(rnp, cpu) {
 			unsigned long bit = leaf_node_cpu_bit(rnp, cpu);
 			if ((rnp->qsmask & bit) != 0) {
-				if (f(per_cpu_ptr(&rcu_data, cpu)))
+				rdp = per_cpu_ptr(&rcu_data, cpu);
+				if (f(rdp)) {
 					mask |= bit;
+					rcu_disable_tick_upon_qs(rdp);
+				}

I am guessing this was the earlier thing you corrected, cool!!

Quoted 54 lines
kernel/rcu/tree.c +6 -6
 			}
 		}
 		if (mask != 0) {
@@ -2322,7 +2343,7 @@ void rcu_force_quiescent_state(void)
 	rnp = __this_cpu_read(rcu_data.mynode);
 	for (; rnp != NULL; rnp = rnp->parent) {
 		ret = (READ_ONCE(rcu_state.gp_flags) & RCU_GP_FLAG_FQS) ||
-		      !raw_spin_trylock(&rnp->fqslock);
+		       !raw_spin_trylock(&rnp->fqslock);
 		if (rnp_old != NULL)
 			raw_spin_unlock(&rnp_old->fqslock);
 		if (ret)
@@ -2855,7 +2876,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp)
 {
 	if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) {
 		rcu_barrier_trace(TPS("LastCB"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		complete(&rcu_state.barrier_completion);
 	} else {
 		rcu_barrier_trace(TPS("CB"), -1, rcu_state.barrier_sequence);
@@ -2879,7 +2900,7 @@ static void rcu_barrier_func(void *unused)
 	} else {
 		debug_rcu_head_unqueue(&rdp->barrier_head);
 		rcu_barrier_trace(TPS("IRQNQ"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 	}
 	rcu_nocb_unlock(rdp);
 }
@@ -2906,7 +2927,7 @@ void rcu_barrier(void)
 	/* Did someone else do our work for us? */
 	if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
 		rcu_barrier_trace(TPS("EarlyExit"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		smp_mb(); /* caller's subsequent code after above check. */
 		mutex_unlock(&rcu_state.barrier_mutex);
 		return;
@@ -2938,11 +2959,11 @@ void rcu_barrier(void)
 			continue;
 		if (rcu_segcblist_n_cbs(&rdp->cblist)) {
 			rcu_barrier_trace(TPS("OnlineQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 			smp_call_function_single(cpu, rcu_barrier_func, NULL, 1);
 		} else {
 			rcu_barrier_trace(TPS("OnlineNQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 		}
 	}
 	put_online_cpus();

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 6 lines
kernel/rcu/tree.c +1 -0
@@ -3168,6 +3189,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

thanks,

- Joel
[snip]

#50 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 59 lines · Joel Fernandes
Quoted 58 lines · Paul E. McKenney
Quoted 43 lines · Frederic Weisbecker
Quoted 29 lines · Paul E. McKenney
Quoted 3 lines · Frederic Weisbecker

On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote:
Looks like it's not the right fix but, should you ever need to set an all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in RCU's CPU-hotplug notifiers. Which does have an effect, especially on the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting purposes? If so, would it make sense for this monitoring to check for long-duration kernel execution and enable the tick in this case? The RCU dyntick machinery can be used to remotely detect the long-duration kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */
Quoted 6 lines

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(), which would be simple wrappers around RCU's private rcu_dynticks_snap() and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner case or three?

Oh I see. Until now we considered than running into the kernel (between user/guest/idle) is supposed to be short but there can be specific places where it doesn't apply.

I'm wondering if, more than just providing wrappers, this shouldn't be entirely driven by RCU using the tick_set_dep_cpu()/tick_clear_dep_cpu() at appropriate timings.

Quoted 8 lines

I don't want to sound like I'm trying to put all the work on you :p It's just that the tick shouldn't know much about RCU, it's rather RCU that is a client for the tick and is probably better suited to determine when a CPU becomes annoying with its extended grace period.

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

Left to itself, RCU would take action only when a given nohz_full in-kernel CPU was delaying a grace period, which is what the (lightly tested) patch below is supposed to help with. If that is all that is needed, well and good!

Quoted 8 lines
Quoted 4 lines

But should we need long-running in-kernel nohz_full CPUs to turn on their ticks when they are not blocking an RCU grace period, for example, when RCU is idle, more will be needed. To that point, isn't there some sort of monitoring that checks up on nohz_full CPUs ever second or so?

Wouldn't such monitoring need to be more often than a second, given that rcu_urgent_qs and rcu_need_heavy_qs are configured typically to be sooner (200-300 jiffies on my system).

Either it would have to be more often than once per second, or RCU would need to retain its more frequent checks. But note that RCU isn't going to check unless there is a grace period in progress.

Quoted 5 lines
Quoted 5 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

Quoted 7 lines
Quoted 4 lines

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Please!

Quoted 69 lines

2 comments below:

Quoted 67 lines

------------------------------------------------------------------------

commit 1cb89508804f6f2fdb79a1be032b1932d52318c4

Author: Paul E. McKenney <[email protected]>
Date:   Mon Aug 12 16:14:00 2019 -0700

rcu: Force tick on for nohz_full CPUs not reaching quiescent states

CPUs running for long time periods in the kernel in nohz_full mode might leave the scheduling-clock interrupt disabled for then full duration of their in-kernel execution. This can (among other things) delay grace periods. This commit therefore forces the tick back on for any nohz_full CPU that is failing to pass through a quiescent state upon return from interrupt, which the resched_cpu() will induce.

kernel/rcu/tree.c +18 -0
index 8c494a692728..8b8f5bffdc5a 100644
@@ -651,6 +651,12 @@ static __always_inline void rcu_nmi_exit_common(bool irq)
 	 */
 	if (rdp->dynticks_nmi_nesting != 1) {
 		trace_rcu_dyntick(TPS("--="), rdp->dynticks_nmi_nesting, rdp->dynticks_nmi_nesting - 2, rdp->dynticks);
+		if (tick_nohz_full_cpu(rdp->cpu) &&
+		    rdp->dynticks_nmi_nesting == 2 &&
+		    rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+			rdp->rcu_forced_tick = true;
+			tick_dep_set_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+		}
 		WRITE_ONCE(rdp->dynticks_nmi_nesting, /* No store tearing. */
 			   rdp->dynticks_nmi_nesting - 2);
 		return;
@@ -886,6 +892,16 @@ void rcu_irq_enter_irqson(void)
 	local_irq_restore(flags);
 }
 
+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+	if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+		tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+}
+
 /**
  * rcu_is_watching - see if RCU thinks that the current CPU is not idle
  *
@@ -1980,6 +1996,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 		if (!offloaded)
 			needwake = rcu_accelerate_cbs(rnp, rdp);
 
+		rcu_disable_tick_upon_qs(rdp);
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);
 		/* ^^^ Released rnp->lock */
 		if (needwake)
@@ -2269,6 +2286,7 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 	int cpu;
 	unsigned long flags;
 	unsigned long mask;
+	struct rcu_data *rdp;
 	struct rcu_node *rnp;
 
Quoted 13 lines

kernel/rcu/tree.c static void force_qs_rnp(int (*f)(struct rcu_data *rdp)) · line 2318

Quoted 11 lines
kernel/rcu/tree.c +4 -1
 	rcu_for_each_leaf_node(rnp) {
@@ -2293,8 +2311,11 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 		for_each_leaf_node_possible_cpu(rnp, cpu) {
 			unsigned long bit = leaf_node_cpu_bit(rnp, cpu);
 			if ((rnp->qsmask & bit) != 0) {
-				if (f(per_cpu_ptr(&rcu_data, cpu)))
+				rdp = per_cpu_ptr(&rcu_data, cpu);
+				if (f(rdp)) {
 					mask |= bit;
+					rcu_disable_tick_upon_qs(rdp);
+				}

I am guessing this was the earlier thing you corrected, cool!!

Like I said, stupid error on my part. The usual kind. ;-)

Quoted 56 lines
Quoted 56 lines
kernel/rcu/tree.c +6 -6
 			}
 		}
 		if (mask != 0) {
@@ -2322,7 +2343,7 @@ void rcu_force_quiescent_state(void)
 	rnp = __this_cpu_read(rcu_data.mynode);
 	for (; rnp != NULL; rnp = rnp->parent) {
 		ret = (READ_ONCE(rcu_state.gp_flags) & RCU_GP_FLAG_FQS) ||
-		      !raw_spin_trylock(&rnp->fqslock);
+		       !raw_spin_trylock(&rnp->fqslock);
 		if (rnp_old != NULL)
 			raw_spin_unlock(&rnp_old->fqslock);
 		if (ret)
@@ -2855,7 +2876,7 @@ static void rcu_barrier_callback(struct rcu_head *rhp)
 {
 	if (atomic_dec_and_test(&rcu_state.barrier_cpu_count)) {
 		rcu_barrier_trace(TPS("LastCB"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		complete(&rcu_state.barrier_completion);
 	} else {
 		rcu_barrier_trace(TPS("CB"), -1, rcu_state.barrier_sequence);
@@ -2879,7 +2900,7 @@ static void rcu_barrier_func(void *unused)
 	} else {
 		debug_rcu_head_unqueue(&rdp->barrier_head);
 		rcu_barrier_trace(TPS("IRQNQ"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 	}
 	rcu_nocb_unlock(rdp);
 }
@@ -2906,7 +2927,7 @@ void rcu_barrier(void)
 	/* Did someone else do our work for us? */
 	if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
 		rcu_barrier_trace(TPS("EarlyExit"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		smp_mb(); /* caller's subsequent code after above check. */
 		mutex_unlock(&rcu_state.barrier_mutex);
 		return;
@@ -2938,11 +2959,11 @@ void rcu_barrier(void)
 			continue;
 		if (rcu_segcblist_n_cbs(&rdp->cblist)) {
 			rcu_barrier_trace(TPS("OnlineQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 			smp_call_function_single(cpu, rcu_barrier_func, NULL, 1);
 		} else {
 			rcu_barrier_trace(TPS("OnlineNQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 		}
 	}
 	put_online_cpus();
@@ -3168,6 +3189,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 8 lines

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 4 lines
kernel/rcu/tree.c +1 -0
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

Well, neither grace periods nor CPU-hotplug operations are atomic, and each can take significant time to complete.

So suppose we have a large system with multiple leaf rcu_node structures (not that 17 CPUs is all that many these days, but please bear with me). Suppose just after a new grace period initializes a given leaf rcu_node structure, one of its CPUs goes offline (yes, that CPU would have to have waited on a grace period, but that might have been the previous grace period). But before the FQS scan notices that RCU is waiting on an offline CPU, the CPU comes back online.

That situation is exactly what the above code is intended to handle.

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

Thanx, Paul

#51 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

On Wed, Aug 14, 2019 at 03:05:16PM -0700, Paul E. McKenney wrote:
[snip]

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 3 lines

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

Left to itself, RCU would take action only when a given nohz_full in-kernel CPU was delaying a grace period, which is what the (lightly tested) patch below is supposed to help with. If that is all that is needed, well and good!

But should we need long-running in-kernel nohz_full CPUs to turn on their ticks when they are not blocking an RCU grace period, for example, when RCU is idle, more will be needed. To that point, isn't there some sort of monitoring that checks up on nohz_full CPUs ever second or so?

Quoted 7 lines
Quoted 3 lines

Wouldn't such monitoring need to be more often than a second, given that rcu_urgent_qs and rcu_need_heavy_qs are configured typically to be sooner (200-300 jiffies on my system).

Either it would have to be more often than once per second, or RCU would need to retain its more frequent checks. But note that RCU isn't going to check unless there is a grace period in progress.

Sure.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Please!

I tried the patch below, but it did not seem to make a difference to the issue I was seeing. My test tree is here in case you can spot anything I did not do right: https://github.com/joelagnel/linux-kernel/commits/rcu/nohz-test
The main patch is here:
https://github.com/joelagnel/linux-kernel/commit/4dc282b559d918a0be826936f997db0bdad7abb3

On the trace output, I grep something like: egrep "(rcu_perf|cpu 3|3d)". I see a few ticks after 300ms, but then there are no more ticks and just a periodic resched_cpu() from rcu_implicit_dynticks_qs():

[   19.534107] rcu_perf-165    12.... 2276436us : rcu_perf_writer: Start of rcuperf test
[   19.557968] rcu_pree-10      0d..1 2287973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.136222] rcu_perf-165     3d.h. 2591894us : rcu_sched_clock_irq: sched-tick
[   20.137185] rcu_perf-165     3d.h2 2591906us : rcu_sched_clock_irq: sched-tick
[   20.138149] rcu_perf-165     3d.h. 2591911us : rcu_sched_clock_irq: sched-tick
[   20.139106] rcu_perf-165     3d.h. 2591915us : rcu_sched_clock_irq: sched-tick
[   20.140077] rcu_perf-165     3d.h. 2591919us : rcu_sched_clock_irq: sched-tick
[   20.141041] rcu_perf-165     3d.h. 2591924us : rcu_sched_clock_irq: sched-tick
[   20.142001] rcu_perf-165     3d.h. 2591928us : rcu_sched_clock_irq: sched-tick
[   20.142961] rcu_perf-165     3d.h. 2591932us : rcu_sched_clock_irq: sched-tick
[   20.143925] rcu_perf-165     3d.h. 2591936us : rcu_sched_clock_irq: sched-tick
[   20.144885] rcu_perf-165     3d.h. 2591940us : rcu_sched_clock_irq: sched-tick
[   20.145876] rcu_perf-165     3d.h. 2591945us : rcu_sched_clock_irq: sched-tick
[   20.146835] rcu_perf-165     3d.h. 2591949us : rcu_sched_clock_irq: sched-tick
[   20.147797] rcu_perf-165     3d.h. 2591953us : rcu_sched_clock_irq: sched-tick
[   20.148759] rcu_perf-165     3d.h. 2591957us : rcu_sched_clock_irq: sched-tick
[   20.151655] rcu_pree-10      0d..1 2591979us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.732938] rcu_pree-10      0d..1 2895960us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   21.318104] rcu_pree-10      0d..1 3199975us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   21.899908] rcu_pree-10      0d..1 3503964us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   22.481316] rcu_pree-10      0d..1 3807990us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   23.065623] rcu_pree-10      0d..1 4111990us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   23.650875] rcu_pree-10      0d..1 4415989us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   24.233999] rcu_pree-10      0d..1 4719978us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   24.818397] rcu_pree-10      0d..1 5023982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   25.402633] rcu_pree-10      0d..1 5327981us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   25.984104] rcu_pree-10      0d..1 5631976us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   26.566100] rcu_pree-10      0d..1 5935982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.144497] rcu_pree-10      0d..1 6239973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.192661] rcu_perf-165     3d.h. 6276923us : rcu_sched_clock_irq: sched-tick
[   27.705789] rcu_pree-10      0d..1 6541901us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.292155] rcu_pree-10      0d..1 6845974us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.874049] rcu_pree-10      0d..1 7149972us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   29.112646] rcu_perf-165     3.... 7275951us : rcu_perf_writer: End of rcuperf test

[snip]

Quoted 46 lines
Quoted 26 lines
Quoted 26 lines
kernel/rcu/tree.c +3 -3
@@ -2906,7 +2927,7 @@ void rcu_barrier(void)
 	/* Did someone else do our work for us? */
 	if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
 		rcu_barrier_trace(TPS("EarlyExit"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		smp_mb(); /* caller's subsequent code after above check. */
 		mutex_unlock(&rcu_state.barrier_mutex);
 		return;
@@ -2938,11 +2959,11 @@ void rcu_barrier(void)
 			continue;
 		if (rcu_segcblist_n_cbs(&rdp->cblist)) {
 			rcu_barrier_trace(TPS("OnlineQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 			smp_call_function_single(cpu, rcu_barrier_func, NULL, 1);
 		} else {
 			rcu_barrier_trace(TPS("OnlineNQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 		}
 	}
 	put_online_cpus();
@@ -3168,6 +3189,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 8 lines

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 4 lines
kernel/rcu/tree.c +1 -0
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

Well, neither grace periods nor CPU-hotplug operations are atomic, and each can take significant time to complete.

So suppose we have a large system with multiple leaf rcu_node structures (not that 17 CPUs is all that many these days, but please bear with me). Suppose just after a new grace period initializes a given leaf rcu_node structure, one of its CPUs goes offline (yes, that CPU would have to have waited on a grace period, but that might have been the previous grace period). But before the FQS scan notices that RCU is waiting on an offline CPU, the CPU comes back online.

Quoted 1 line

That situation is exactly what the above code is intended to handle.

That makes sense!

Quoted 3 lines

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

I did not fully understand the question in brackets though, a task can be on a different CPU though which has nothing to do with the CPU that's going offline/online so it could totally be waiting on a grace period right?

Also waiting on a grace period that hasn't even started is totally possible:

GP1 GP2

|<--------->|<-------->|
     ^                 ^
     |                 |____  task gets unblocked

task blocks
on synchronize_rcu
but is waiting on
GP2 which hasn't started

Or did I misunderstand the question?

thanks!

- Joel

#52 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 39 lines · Joel Fernandes

On Wed, Aug 14, 2019 at 03:05:16PM -0700, Paul E. McKenney wrote:
[snip]

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 3 lines

Arming a CPU timer could also be an alternative to tick_set_dep_cpu() for that.

What do you think?

Left to itself, RCU would take action only when a given nohz_full in-kernel CPU was delaying a grace period, which is what the (lightly tested) patch below is supposed to help with. If that is all that is needed, well and good!

But should we need long-running in-kernel nohz_full CPUs to turn on their ticks when they are not blocking an RCU grace period, for example, when RCU is idle, more will be needed. To that point, isn't there some sort of monitoring that checks up on nohz_full CPUs ever second or so?

Quoted 7 lines
Quoted 3 lines

Wouldn't such monitoring need to be more often than a second, given that rcu_urgent_qs and rcu_need_heavy_qs are configured typically to be sooner (200-300 jiffies on my system).

Either it would have to be more often than once per second, or RCU would need to retain its more frequent checks. But note that RCU isn't going to check unless there is a grace period in progress.

Sure.

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Quoted 7 lines
Quoted 1 line

Please!

I tried the patch below, but it did not seem to make a difference to the issue I was seeing. My test tree is here in case you can spot anything I did not do right: https://github.com/joelagnel/linux-kernel/commits/rcu/nohz-test
The main patch is here:
https://github.com/joelagnel/linux-kernel/commit/4dc282b559d918a0be826936f997db0bdad7abb3

That is more aggressive that rcutorture's rcu_torture_fwd_prog_nr(), so I am guessing that I need to up rcu_torture_fwd_prog_nr()'s game. I am currently testing that.

Quoted 30 lines

On the trace output, I grep something like: egrep "(rcu_perf|cpu 3|3d)". I see a few ticks after 300ms, but then there are no more ticks and just a periodic resched_cpu() from rcu_implicit_dynticks_qs():

[   19.534107] rcu_perf-165    12.... 2276436us : rcu_perf_writer: Start of rcuperf test
[   19.557968] rcu_pree-10      0d..1 2287973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.136222] rcu_perf-165     3d.h. 2591894us : rcu_sched_clock_irq: sched-tick
[   20.137185] rcu_perf-165     3d.h2 2591906us : rcu_sched_clock_irq: sched-tick
[   20.138149] rcu_perf-165     3d.h. 2591911us : rcu_sched_clock_irq: sched-tick
[   20.139106] rcu_perf-165     3d.h. 2591915us : rcu_sched_clock_irq: sched-tick
[   20.140077] rcu_perf-165     3d.h. 2591919us : rcu_sched_clock_irq: sched-tick
[   20.141041] rcu_perf-165     3d.h. 2591924us : rcu_sched_clock_irq: sched-tick
[   20.142001] rcu_perf-165     3d.h. 2591928us : rcu_sched_clock_irq: sched-tick
[   20.142961] rcu_perf-165     3d.h. 2591932us : rcu_sched_clock_irq: sched-tick
[   20.143925] rcu_perf-165     3d.h. 2591936us : rcu_sched_clock_irq: sched-tick
[   20.144885] rcu_perf-165     3d.h. 2591940us : rcu_sched_clock_irq: sched-tick
[   20.145876] rcu_perf-165     3d.h. 2591945us : rcu_sched_clock_irq: sched-tick
[   20.146835] rcu_perf-165     3d.h. 2591949us : rcu_sched_clock_irq: sched-tick
[   20.147797] rcu_perf-165     3d.h. 2591953us : rcu_sched_clock_irq: sched-tick
[   20.148759] rcu_perf-165     3d.h. 2591957us : rcu_sched_clock_irq: sched-tick
[   20.151655] rcu_pree-10      0d..1 2591979us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.732938] rcu_pree-10      0d..1 2895960us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   21.318104] rcu_pree-10      0d..1 3199975us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   21.899908] rcu_pree-10      0d..1 3503964us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   22.481316] rcu_pree-10      0d..1 3807990us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   23.065623] rcu_pree-10      0d..1 4111990us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   23.650875] rcu_pree-10      0d..1 4415989us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   24.233999] rcu_pree-10      0d..1 4719978us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   24.818397] rcu_pree-10      0d..1 5023982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   25.402633] rcu_pree-10      0d..1 5327981us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
Quoted 8 lines
[   25.984104] rcu_pree-10      0d..1 5631976us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   26.566100] rcu_pree-10      0d..1 5935982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.144497] rcu_pree-10      0d..1 6239973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.192661] rcu_perf-165     3d.h. 6276923us : rcu_sched_clock_irq: sched-tick
[   27.705789] rcu_pree-10      0d..1 6541901us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.292155] rcu_pree-10      0d..1 6845974us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.874049] rcu_pree-10      0d..1 7149972us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   29.112646] rcu_perf-165     3.... 7275951us : rcu_perf_writer: End of rcuperf test

That would be due to my own stupidity. I forgot to clear ->rcu_forced_tick in rcu_disable_tick_upon_qs() inside the "if" statement. This of course prevents rcu_nmi_exit_common() from ever re-enabling it.

Excellent catch! Thank you for testing this!!!

Quoted 65 lines

[snip]

Quoted 46 lines
Quoted 26 lines
Quoted 26 lines
kernel/rcu/tree.c +3 -3
@@ -2906,7 +2927,7 @@ void rcu_barrier(void)
 	/* Did someone else do our work for us? */
 	if (rcu_seq_done(&rcu_state.barrier_sequence, s)) {
 		rcu_barrier_trace(TPS("EarlyExit"), -1,
-				   rcu_state.barrier_sequence);
+				  rcu_state.barrier_sequence);
 		smp_mb(); /* caller's subsequent code after above check. */
 		mutex_unlock(&rcu_state.barrier_mutex);
 		return;
@@ -2938,11 +2959,11 @@ void rcu_barrier(void)
 			continue;
 		if (rcu_segcblist_n_cbs(&rdp->cblist)) {
 			rcu_barrier_trace(TPS("OnlineQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 			smp_call_function_single(cpu, rcu_barrier_func, NULL, 1);
 		} else {
 			rcu_barrier_trace(TPS("OnlineNQ"), cpu,
-					   rcu_state.barrier_sequence);
+					  rcu_state.barrier_sequence);
 		}
 	}
 	put_online_cpus();
@@ -3168,6 +3189,7 @@ void rcu_cpu_starting(unsigned int cpu)
 	rdp->rcu_onl_gp_seq = READ_ONCE(rcu_state.gp_seq);
 	rdp->rcu_onl_gp_flags = READ_ONCE(rcu_state.gp_flags);
Quoted 8 lines

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 4 lines
kernel/rcu/tree.c +1 -0
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

Well, neither grace periods nor CPU-hotplug operations are atomic, and each can take significant time to complete.

So suppose we have a large system with multiple leaf rcu_node structures (not that 17 CPUs is all that many these days, but please bear with me). Suppose just after a new grace period initializes a given leaf rcu_node structure, one of its CPUs goes offline (yes, that CPU would have to have waited on a grace period, but that might have been the previous grace period). But before the FQS scan notices that RCU is waiting on an offline CPU, the CPU comes back online.

Quoted 1 line

That situation is exactly what the above code is intended to handle.

That makes sense!

Quoted 3 lines

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

I did not fully understand the question in brackets though, a task can be on a different CPU though which has nothing to do with the CPU that's going offline/online so it could totally be waiting on a grace period right?

Also waiting on a grace period that hasn't even started is totally possible:

GP1 GP2

|<--------->|<-------->|
     ^                 ^
     |                 |____  task gets unblocked
Quoted 6 lines

task blocks
on synchronize_rcu
but is waiting on
GP2 which hasn't started

Or did I misunderstand the question?

There is a ->gp_tasks field in the leaf rcu_node structures that references a list of tasks blocking the current grace period. When there is no grace period in progress (as is the case from the end of GP1 to the beginning of GP2, the RCU code expects ->gp_tasks to be NULL. Without the curiosity code you pointed out above, ->gp_tasks could in fact end up being non-NULL when no grace period was in progress.

And did end up being non-NULL from time to time, initially every few hundred hours of a particular rcutorture scenario.

Thanx, Paul

#53 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 38 lines · Paul E. McKenney
Quoted 15 lines · Joel Fernandes

On Wed, Aug 14, 2019 at 03:05:16PM -0700, Paul E. McKenney wrote:
[snip]

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Quoted 7 lines
Quoted 1 line

Please!

I tried the patch below, but it did not seem to make a difference to the issue I was seeing. My test tree is here in case you can spot anything I did not do right: https://github.com/joelagnel/linux-kernel/commits/rcu/nohz-test
The main patch is here:
https://github.com/joelagnel/linux-kernel/commit/4dc282b559d918a0be826936f997db0bdad7abb3

That is more aggressive that rcutorture's rcu_torture_fwd_prog_nr(), so I am guessing that I need to up rcu_torture_fwd_prog_nr()'s game. I am currently testing that.

Quoted 10 lines

On the trace output, I grep something like: egrep "(rcu_perf|cpu 3|3d)". I see a few ticks after 300ms, but then there are no more ticks and just a periodic resched_cpu() from rcu_implicit_dynticks_qs():

[   19.534107] rcu_perf-165    12.... 2276436us : rcu_perf_writer: Start of rcuperf test
[   19.557968] rcu_pree-10      0d..1 2287973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.136222] rcu_perf-165     3d.h. 2591894us : rcu_sched_clock_irq: sched-tick
[   20.137185] rcu_perf-165     3d.h2 2591906us : rcu_sched_clock_irq: sched-tick
[   20.138149] rcu_perf-165     3d.h. 2591911us : rcu_sched_clock_irq: sched-tick
[   20.139106] rcu_perf-165     3d.h. 2591915us : rcu_sched_clock_irq: sched-tick

[snip]

Quoted 4 lines
Quoted 4 lines
[   20.147797] rcu_perf-165     3d.h. 2591953us : rcu_sched_clock_irq: sched-tick
[   20.148759] rcu_perf-165     3d.h. 2591957us : rcu_sched_clock_irq: sched-tick
[   20.151655] rcu_pree-10      0d..1 2591979us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.732938] rcu_pree-10      0d..1 2895960us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3

[snip]

Quoted 8 lines
Quoted 7 lines
[   26.566100] rcu_pree-10      0d..1 5935982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.144497] rcu_pree-10      0d..1 6239973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.192661] rcu_perf-165     3d.h. 6276923us : rcu_sched_clock_irq: sched-tick
[   27.705789] rcu_pree-10      0d..1 6541901us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.292155] rcu_pree-10      0d..1 6845974us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.874049] rcu_pree-10      0d..1 7149972us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   29.112646] rcu_perf-165     3.... 7275951us : rcu_perf_writer: End of rcuperf test
Quoted 5 lines

That would be due to my own stupidity. I forgot to clear ->rcu_forced_tick in rcu_disable_tick_upon_qs() inside the "if" statement. This of course prevents rcu_nmi_exit_common() from ever re-enabling it.

Excellent catch! Thank you for testing this!!!

Ah I missed it too. Happy to help! I tried setting it as below but getting same results:

+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+ if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+ tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+ rdp->rcu_forced_tick = false;
+}
+

Quoted 53 lines
Quoted 39 lines

[snip]

Quoted 20 lines
Quoted 8 lines

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 4 lines
kernel/rcu/tree.c +1 -0
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

Well, neither grace periods nor CPU-hotplug operations are atomic, and each can take significant time to complete.

So suppose we have a large system with multiple leaf rcu_node structures (not that 17 CPUs is all that many these days, but please bear with me). Suppose just after a new grace period initializes a given leaf rcu_node structure, one of its CPUs goes offline (yes, that CPU would have to have waited on a grace period, but that might have been the previous grace period). But before the FQS scan notices that RCU is waiting on an offline CPU, the CPU comes back online.

Quoted 1 line

That situation is exactly what the above code is intended to handle.

That makes sense!

Quoted 3 lines

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

I did not fully understand the question in brackets though, a task can be on a different CPU though which has nothing to do with the CPU that's going offline/online so it could totally be waiting on a grace period right?

Also waiting on a grace period that hasn't even started is totally possible:

GP1 GP2

|<--------->|<-------->|
     ^                 ^
     |                 |____  task gets unblocked
Quoted 6 lines

task blocks
on synchronize_rcu
but is waiting on
GP2 which hasn't started

Or did I misunderstand the question?

There is a ->gp_tasks field in the leaf rcu_node structures that references a list of tasks blocking the current grace period. When there is no grace period in progress (as is the case from the end of GP1 to the beginning of GP2, the RCU code expects ->gp_tasks to be NULL. Without the curiosity code you pointed out above, ->gp_tasks could in fact end up being non-NULL when no grace period was in progress.

Quoted 2 lines

And did end up being non-NULL from time to time, initially every few hundred hours of a particular rcutorture scenario.

Oh ok! I will think more about it. I am not yet able to connect the gp_tasks being non-NULL to the CPU going offline/online scenario though. Maybe I should delete this code, run an experiment and trace for this condition (gp_tasks != NULL)?

I love it how you found these issues by heavy testing and fixed them.

thanks,

- Joel

#54 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 62 lines · Joel Fernandes
Quoted 38 lines · Paul E. McKenney
Quoted 15 lines · Joel Fernandes

On Wed, Aug 14, 2019 at 03:05:16PM -0700, Paul E. McKenney wrote:
[snip]

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Quoted 7 lines
Quoted 1 line

Please!

I tried the patch below, but it did not seem to make a difference to the issue I was seeing. My test tree is here in case you can spot anything I did not do right: https://github.com/joelagnel/linux-kernel/commits/rcu/nohz-test
The main patch is here:
https://github.com/joelagnel/linux-kernel/commit/4dc282b559d918a0be826936f997db0bdad7abb3

That is more aggressive that rcutorture's rcu_torture_fwd_prog_nr(), so I am guessing that I need to up rcu_torture_fwd_prog_nr()'s game. I am currently testing that.

Quoted 10 lines

On the trace output, I grep something like: egrep "(rcu_perf|cpu 3|3d)". I see a few ticks after 300ms, but then there are no more ticks and just a periodic resched_cpu() from rcu_implicit_dynticks_qs():

[   19.534107] rcu_perf-165    12.... 2276436us : rcu_perf_writer: Start of rcuperf test
[   19.557968] rcu_pree-10      0d..1 2287973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.136222] rcu_perf-165     3d.h. 2591894us : rcu_sched_clock_irq: sched-tick
[   20.137185] rcu_perf-165     3d.h2 2591906us : rcu_sched_clock_irq: sched-tick
[   20.138149] rcu_perf-165     3d.h. 2591911us : rcu_sched_clock_irq: sched-tick
[   20.139106] rcu_perf-165     3d.h. 2591915us : rcu_sched_clock_irq: sched-tick

[snip]

Quoted 4 lines
Quoted 4 lines
[   20.147797] rcu_perf-165     3d.h. 2591953us : rcu_sched_clock_irq: sched-tick
[   20.148759] rcu_perf-165     3d.h. 2591957us : rcu_sched_clock_irq: sched-tick
[   20.151655] rcu_pree-10      0d..1 2591979us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.732938] rcu_pree-10      0d..1 2895960us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3

[snip]

Quoted 8 lines
Quoted 7 lines
[   26.566100] rcu_pree-10      0d..1 5935982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.144497] rcu_pree-10      0d..1 6239973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.192661] rcu_perf-165     3d.h. 6276923us : rcu_sched_clock_irq: sched-tick
[   27.705789] rcu_pree-10      0d..1 6541901us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.292155] rcu_pree-10      0d..1 6845974us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.874049] rcu_pree-10      0d..1 7149972us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   29.112646] rcu_perf-165     3.... 7275951us : rcu_perf_writer: End of rcuperf test
Quoted 5 lines

That would be due to my own stupidity. I forgot to clear ->rcu_forced_tick in rcu_disable_tick_upon_qs() inside the "if" statement. This of course prevents rcu_nmi_exit_common() from ever re-enabling it.

Excellent catch! Thank you for testing this!!!

Ah I missed it too. Happy to help! I tried setting it as below but getting same results:

kernel/rcu/tree.c void rcu_irq_enter_irqson(void)

Quoted 9 lines
kernel/rcu/tree.c +9 -0
+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+       if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+               tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+       rdp->rcu_forced_tick = false;

I put this inside the "if" statement, though I would not expect that to change behavior in this case.

Does your test case still avoid turning on the tick more than once? Or is it turning on the tick each time the grace period gets too long, but without the tick managing to end the grace periods?

Quoted 56 lines
kernel/rcu/tree.c +2 -0
+}
+
Quoted 54 lines
Quoted 40 lines

[snip]

Quoted 20 lines
Quoted 8 lines

kernel/rcu/tree.c void rcu_cpu_starting(unsigned int cpu) · line 3194

Quoted 4 lines
kernel/rcu/tree.c +1 -0
 	if (rnp->qsmask & mask) { /* RCU waiting on incoming CPU? */
+		rcu_disable_tick_upon_qs(rdp);
 		/* Report QS -after- changing ->qsmaskinitnext! */
 		rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags);

Just curious about the existing code. If a CPU is just starting up (after bringing it online), how can RCU be waiting on it? I thought RCU would not be watching offline CPUs.

Well, neither grace periods nor CPU-hotplug operations are atomic, and each can take significant time to complete.

So suppose we have a large system with multiple leaf rcu_node structures (not that 17 CPUs is all that many these days, but please bear with me). Suppose just after a new grace period initializes a given leaf rcu_node structure, one of its CPUs goes offline (yes, that CPU would have to have waited on a grace period, but that might have been the previous grace period). But before the FQS scan notices that RCU is waiting on an offline CPU, the CPU comes back online.

Quoted 1 line

That situation is exactly what the above code is intended to handle.

That makes sense!

Quoted 3 lines

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

I did not fully understand the question in brackets though, a task can be on a different CPU though which has nothing to do with the CPU that's going offline/online so it could totally be waiting on a grace period right?

Also waiting on a grace period that hasn't even started is totally possible:

GP1 GP2

|<--------->|<-------->|
     ^                 ^
     |                 |____  task gets unblocked
Quoted 6 lines

task blocks
on synchronize_rcu
but is waiting on
GP2 which hasn't started

Or did I misunderstand the question?

There is a ->gp_tasks field in the leaf rcu_node structures that references a list of tasks blocking the current grace period. When there is no grace period in progress (as is the case from the end of GP1 to the beginning of GP2, the RCU code expects ->gp_tasks to be NULL. Without the curiosity code you pointed out above, ->gp_tasks could in fact end up being non-NULL when no grace period was in progress.

Quoted 7 lines
Quoted 2 lines

And did end up being non-NULL from time to time, initially every few hundred hours of a particular rcutorture scenario.

Oh ok! I will think more about it. I am not yet able to connect the gp_tasks being non-NULL to the CPU going offline/online scenario though. Maybe I should delete this code, run an experiment and trace for this condition (gp_tasks != NULL)?

Or you could dig through the git logs for this code change.

Quoted 1 line

I love it how you found these issues by heavy testing and fixed them.

Me, I would have rather foreseen them and avoided them in the first place, but I agree that it is better for rcutorture to find them than for some hapless user somewhere to be inconvenienced by them. ;-)

Thanx, Paul

#55 lore raw

Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

Quoted 74 lines · Paul E. McKenney
Quoted 60 lines · Joel Fernandes
Quoted 36 lines · Paul E. McKenney
Quoted 14 lines

[snip]

Quoted 13 lines
Quoted 12 lines
Quoted 9 lines

If so, perhaps that monitoring could periodically invoke an RCU function that I provide for deciding when to turn the tick on. We would also need to work out how to turn the tick off in a timely fashion once the CPU got out of kernel mode, perhaps in rcu_user_enter() or rcu_nmi_exit_common().

If this would be called only every second or so, the separate grace-period checking is still needed for its shorter timespan, though.

Thoughts?

Do you want me to test the below patch to see if it fixes the issue with my other test case (where I had a nohz full CPU holding up a grace period).

Quoted 7 lines
Quoted 1 line

Please!

I tried the patch below, but it did not seem to make a difference to the issue I was seeing. My test tree is here in case you can spot anything I did not do right: https://github.com/joelagnel/linux-kernel/commits/rcu/nohz-test
The main patch is here:
https://github.com/joelagnel/linux-kernel/commit/4dc282b559d918a0be826936f997db0bdad7abb3

That is more aggressive that rcutorture's rcu_torture_fwd_prog_nr(), so I am guessing that I need to up rcu_torture_fwd_prog_nr()'s game. I am currently testing that.

Quoted 10 lines

On the trace output, I grep something like: egrep "(rcu_perf|cpu 3|3d)". I see a few ticks after 300ms, but then there are no more ticks and just a periodic resched_cpu() from rcu_implicit_dynticks_qs():

[   19.534107] rcu_perf-165    12.... 2276436us : rcu_perf_writer: Start of rcuperf test
[   19.557968] rcu_pree-10      0d..1 2287973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.136222] rcu_perf-165     3d.h. 2591894us : rcu_sched_clock_irq: sched-tick
[   20.137185] rcu_perf-165     3d.h2 2591906us : rcu_sched_clock_irq: sched-tick
[   20.138149] rcu_perf-165     3d.h. 2591911us : rcu_sched_clock_irq: sched-tick
[   20.139106] rcu_perf-165     3d.h. 2591915us : rcu_sched_clock_irq: sched-tick

[snip]

Quoted 4 lines
Quoted 4 lines
[   20.147797] rcu_perf-165     3d.h. 2591953us : rcu_sched_clock_irq: sched-tick
[   20.148759] rcu_perf-165     3d.h. 2591957us : rcu_sched_clock_irq: sched-tick
[   20.151655] rcu_pree-10      0d..1 2591979us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   20.732938] rcu_pree-10      0d..1 2895960us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3

[snip]

Quoted 8 lines
Quoted 7 lines
[   26.566100] rcu_pree-10      0d..1 5935982us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.144497] rcu_pree-10      0d..1 6239973us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   27.192661] rcu_perf-165     3d.h. 6276923us : rcu_sched_clock_irq: sched-tick
[   27.705789] rcu_pree-10      0d..1 6541901us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.292155] rcu_pree-10      0d..1 6845974us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   28.874049] rcu_pree-10      0d..1 7149972us : rcu_implicit_dynticks_qs: Sending urgent resched to cpu 3
[   29.112646] rcu_perf-165     3.... 7275951us : rcu_perf_writer: End of rcuperf test
Quoted 5 lines

That would be due to my own stupidity. I forgot to clear ->rcu_forced_tick in rcu_disable_tick_upon_qs() inside the "if" statement. This of course prevents rcu_nmi_exit_common() from ever re-enabling it.

Excellent catch! Thank you for testing this!!!

Ah I missed it too. Happy to help! I tried setting it as below but getting same results:

Review

Quoted 9 lines
(quoted diff) +9 -0
+/*
+ * If the scheduler-clock interrupt was enabled on a nohz_full CPU
+ * in order to get to a quiescent state, disable it.
+ */
+void rcu_disable_tick_upon_qs(struct rcu_data *rdp)
+{
+       if (tick_nohz_full_cpu(rdp->cpu) && rdp->rcu_forced_tick)
+               tick_dep_clear_cpu(rdp->cpu, TICK_DEP_MASK_RCU);
+       rdp->rcu_forced_tick = false;

I put this inside the "if" statement, though I would not expect that to change behavior in this case.

Quoted 3 lines

Does your test case still avoid turning on the tick more than once? Or is it turning on the tick each time the grace period gets too long, but without the tick managing to end the grace periods?

I will put some more prints and let you know. But it looks like I see a print from rcu_sched_clock_irq() only once at around 700ms from the start of the test loop. After that I don't see prints at all for the rest of the 7 seconds of the test.

Before the test starts, I see several rcu_sched_clock_irq() at the regular tick interval of 1 ms (HZ=1000).

Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 15 lines

[snip]

Quoted 3 lines

Without that code, RCU can give false-positive splats at various points in its processing. ("Wait! How can a task be blocked waiting on a grace period that hasn't even started yet???")

I did not fully understand the question in brackets though, a task can be on a different CPU though which has nothing to do with the CPU that's going offline/online so it could totally be waiting on a grace period right?

Also waiting on a grace period that hasn't even started is totally possible:

GP1 GP2

|<--------->|<-------->|
     ^                 ^
     |                 |____  task gets unblocked
Quoted 6 lines

task blocks
on synchronize_rcu
but is waiting on
GP2 which hasn't started

Or did I misunderstand the question?

There is a ->gp_tasks field in the leaf rcu_node structures that references a list of tasks blocking the current grace period. When there is no grace period in progress (as is the case from the end of GP1 to the beginning of GP2, the RCU code expects ->gp_tasks to be NULL. Without the curiosity code you pointed out above, ->gp_tasks could in fact end up being non-NULL when no grace period was in progress.

And did end up being non-NULL from time to time, initially every few hundred hours of a particular rcutorture scenario.

Quoted 6 lines
Quoted 4 lines

Oh ok! I will think more about it. I am not yet able to connect the gp_tasks being non-NULL to the CPU going offline/online scenario though. Maybe I should delete this code, run an experiment and trace for this condition (gp_tasks != NULL)?

Or you could dig through the git logs for this code change.

Ok will do.

Quoted 5 lines
Quoted 1 line

I love it how you found these issues by heavy testing and fixed them.

Me, I would have rather foreseen them and avoided them in the first place, but I agree that it is better for rcutorture to find them than for some hapless user somewhere to be inconvenienced by them. ;-)

True, forseeing is always better ;)

thanks,

- Joel