Skip to contentSkip to conversation

Real-time elimination of RCU_SOFTIRQ

RFC PATCH Superseded Paul E. McKenney 2 participants 11 messages Mar 29, 2019 – Apr 4, 2019

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Real-time elimination of RCU_SOFTIRQ

Hello!

This series is an early posting of code to add a boot-time option to move RCU's softirq processing to per-CPU kthreads. This is not done by default for performance reasons, nor are these reasons theoretical. In fact, earlier attempts to do just this were not met with silence.

The patches are as follows:

1. Enable elimination of Tree-RCU softirq processing via a new
rcutree.use_softirq kernel boot parameter. This defaults to 1, so boot with "rcutree.use_softirq=0" to move RCU_SOFTIRQ work to the rcuc kthreads. Courtesy of Sebastian Andrzej Siewior.

2. Improve wakeup-safety checks in rcu_read_unlock_special(),
thus allowing both rcuc kthreads and reasonably snappy expedited RCU grace periods.

Thanx, Paul

------------------------------------------------------------------------

 Documentation/admin-guide/kernel-parameters.txt |    6 
 include/linux/sched.h                           |    2 
 kernel/rcu/tree.c                               |  138 +++++++++++++++++++--
 kernel/rcu/tree.h                               |    2 
 kernel/rcu/tree_plugin.h                        |  153 ++++--------------------
 5 files changed, 161 insertions(+), 140 deletions(-)
#2 lore raw

PATCH 2/2 rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

When RCU core processing is offloaded from RCU_SOFTIRQ to the rcuc kthreads, a full and unconditional wakeup is required to initiate RCU core processing. In contrast, when RCU core processing is carried out by RCU_SOFTIRQ, a raise_softirq() suffices. Of course, there are situations where raise_softirq() does a full wakeup, but these do not occur with normal usage of rcu_read_unlock().

The reason that full wakeups can be problematic is that the scheduler sometimes invokes rcu_read_unlock() with its pi or rq locks held, which can of course result in deadlock in CONFIG_PREEMPT=y kernels when rcu_read_unlock() invokes the scheduler. Scheduler invocations can happen in the following situations: (1) The just-ended reader has been subjected to RCU priority boosting, in which case rcu_read_unlock() must deboost,
(2) Interrupts were disabled across the call to rcu_read_unlock(), so the quiescent state must be deferred, requiring a wakeup of the rcuc kthread corresponding to the current CPU.

Now, the scheduler may hold one of its locks across rcu_read_unlock() only if preemption has been disabled across the entire RCU read-side critical section, which in the days prior to RCU flavor consolidation meant that rcu_read_unlock() never needed to do wakeups. However, this is no longer the case for any but the first rcu_read_unlock() following a condition (e.g., preempted RCU reader) requiring special rcu_read_unlock() attention. For example, an RCU read-side critical section might be preempted, but preemption might be disabled across the rcu_read_unlock(). The rcu_read_unlock() must defer the quiescent state, and therefore leaves the task queued on its leaf rcu_node structure. If a scheduler interrupt occurs, the scheduler might well invoke rcu_read_unlock() with one of its locks held. However, the preempted task is still queued, so rcu_read_unlock() will attempt to defer the quiescent state once more. When RCU core processing is carried out by RCU_SOFTIRQ, this works just
fine: The raise_softirq() function simply sets a bit in a per-CPU mask and the RCU core processing will be undertaken upon return from interrupt.

Not so when RCU core processing is carried out by the rcuc kthread: In this case, the required wakeup can result in deadlock.

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This commit therefore introduces a ->deferred_qs flag to the task_struct structure's rcu_special structure. This flag is initially false, and is set to true by the first call to rcu_read_unlock() requiring special attention, then finally reset back to false when the quiescent state is finally reported. Then rcu_read_unlock() attempts full wakeups only when ->deferred_qs is false, that is, on the first rcu_read_unlock() requiring special attention. Note that a chain of RCU readers linked by some other sort of reader may find that a later rcu_read_unlock() is once again able to do a full wakeup, courtesy of an intervening preemption:

	rcu_read_lock();
	/* preempted */
	local_irq_disable();
	rcu_read_unlock(); /* Can do full wakeup, sets ->deferred_qs. */
	rcu_read_lock();
	local_irq_enable();
	preempt_disable()
	rcu_read_unlock(); /* Cannot do full wakeup, ->deferred_qs set. */
	rcu_read_lock();
	preempt_enable();
	/* preempted, >deferred_qs reset. */
	local_irq_disable();
	rcu_read_unlock(); /* Can again do full wakeup, sets ->deferred_qs. */

Such linked RCU readers do not yet seem to appear in the Linux kernel, and it is probably best if they don't. However, RCU needs to handle them, and some variations on this theme could make even raise_softirq() unsafe due to the possibility of its doing a full wakeup. This commit therefore also avoids invoking raise_softirq() when the ->deferred_qs set flag is set.

Cc Sebastian Andrzej Siewior <[email protected]>
---
 include/linux/sched.h    |  2 +-
 kernel/rcu/tree_plugin.h | 19 ++++++++++++++-----
 2 files changed, 15 insertions(+), 6 deletions(-)
include/linux/sched.h +1 -1
index 1549584a1538..3164b6798fe5 100644
@@ -566,7 +566,7 @@ union rcu_special {
 		u8			blocked;
 		u8			need_qs;
 		u8			exp_hint; /* Hint for performance. */
-		u8			pad; /* No garbage from compiler! */
+		u8			deferred_qs;
 	} b; /* Bits. */
 	u32 s; /* Set of bits. */
 };
kernel/rcu/tree_plugin.h +14 -5
index 21611862e083..75110ea75d01 100644
@@ -455,6 +455,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
 		local_irq_restore(flags);
 		return;
 	}
+	t->rcu_read_unlock_special.b.deferred_qs = false;
 	if (special.b.need_qs) {
 		rcu_qs();
 		t->rcu_read_unlock_special.b.need_qs = false;
@@ -605,16 +606,24 @@ static void rcu_read_unlock_special(struct task_struct *t)
 	local_irq_save(flags);
 	irqs_were_disabled = irqs_disabled_flags(flags);
 	if (preempt_bh_were_disabled || irqs_were_disabled) {
-		WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false);
-		/* Need to defer quiescent state until everything is enabled. */
-		if (irqs_were_disabled && use_softirq) {
-			/* Enabling irqs does not reschedule, so... */
+		t->rcu_read_unlock_special.b.exp_hint = false;
+		// Need to defer quiescent state until everything is enabled.
+		if (irqs_were_disabled && use_softirq &&
+		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
+			// Using softirq, safe to awaken, and we get
+			// no help from enabling irqs, unlike bh/preempt.
 			raise_softirq_irqoff(RCU_SOFTIRQ);
+		} else if (irqs_were_disabled && !use_softirq &&
+			   !t->rcu_read_unlock_special.b.deferred_qs) {
+			// Safe to awaken and we get no help from enabling
+			// irqs, unlike bh/preempt.
+			invoke_rcu_core();
 		} else {
-			/* Enabling BH or preempt does reschedule, so... */
+			// Enabling BH or preempt does reschedule, so...
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
 		}
+		t->rcu_read_unlock_special.b.deferred_qs = true;
 		local_irq_restore(flags);
 		return;
 	}
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/2 rcu: Enable elimination of Tree-RCU softirq processing

In reply to Paul E. McKenney

From: Sebastian Andrzej Siewior <[email protected]>

Some workloads need to change kthread priority for RCU core processing without affecting other softirq work. This commit therefore introduces the rcutree.use_softirq kernel boot parameter, which moves the RCU core work from softirq to a per-CPU SCHED_OTHER kthread named rcuc. Use of SCHED_OTHER approach avoids the scalability problems that appeared with the earlier attempt to move RCU core processing to from softirq to kthreads. That said, kernels built with RCU_BOOST=y will run the rcuc kthreads at the RCU-boosting priority.

Note that rcutree.use_softirq=0 must be specified to move RCU core processing to the rcuc kthreads: rcutree.use_softirq=1 is the default.

[ paulmck: Adjust for invoke_rcu_callbacks() only ever being invoked
from RCU core processing, in contrast to softirq->rcuc transition in old mainline RCU priority boosting. ]
[ paulmck: Avoid wakeups when scheduler might have invoked rcu_read_unlock()
while holding rq or pi locks, also possibly fixing a pre-existing latent bug involving raise_softirq()-induced wakeups. ]

---
 .../admin-guide/kernel-parameters.txt         |   6 +
 kernel/rcu/tree.c                             | 138 ++++++++++++++++--
 kernel/rcu/tree.h                             |   2 +-
 kernel/rcu/tree_plugin.h                      | 134 ++---------------
 4 files changed, 146 insertions(+), 134 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +6 -0
index d377a2166b79..e2ffb1d9de03 100644
@@ -3672,6 +3672,12 @@
 			the propagation of recent CPU-hotplug changes up
 			the rcu_node combining tree.
 
+	rcutree.use_softirq=	[KNL]
+			If set to zero, move all RCU_SOFTIRQ processing to
+			per-CPU rcuc kthreads.  Defaults to a non-zero
+			value, meaning that RCU_SOFTIRQ is used by default.
+			Specify rcutree.use_softirq=0 to use rcuc kthreads.
+
 	rcutree.rcu_fanout_exact= [KNL]
 			Disable autobalancing of the rcu_node combining
 			tree.  This is used by rcutorture, and might
kernel/rcu/tree.c +125 -13

180 lines not on this page · raw message

kernel/rcu/tree.h +1 -1
index e253d11af3c4..a1a72a1ecb02 100644
@@ -407,8 +407,8 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func);
 static void dump_blkd_tasks(struct rcu_node *rnp, int ncheck);
 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags);
 static void rcu_preempt_boost_start_gp(struct rcu_node *rnp);
-static void invoke_rcu_callbacks_kthread(void);
 static bool rcu_is_callbacks_kthread(void);
+static void rcu_cpu_kthread_setup(unsigned int cpu);
 static void __init rcu_spawn_boost_kthreads(void);
 static void rcu_prepare_kthreads(int cpu);
 static void rcu_cleanup_after_idle(void);
kernel/rcu/tree_plugin.h +14 -120

194 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#3 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

In reply to Paul E. McKenney
Quoted 6 lines · Paul E. McKenney

When RCU core processing is offloaded from RCU_SOFTIRQ to the rcuc kthreads, a full and unconditional wakeup is required to initiate RCU core processing. In contrast, when RCU core processing is carried out by RCU_SOFTIRQ, a raise_softirq() suffices. Of course, there are situations where raise_softirq() does a full wakeup, but these do not occur with normal usage of rcu_read_unlock().

Do we have a comment somewhere explaining why?

Quoted 6 lines

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This is all PREEMPT=y kernels, right? Where is the latency coming from? Because PREEMPT=y _should_ react quite quickly.

Quoted 10 lines

This commit therefore introduces a ->deferred_qs flag to the task_struct structure's rcu_special structure. This flag is initially false, and is set to true by the first call to rcu_read_unlock() requiring special attention, then finally reset back to false when the quiescent state is finally reported. Then rcu_read_unlock() attempts full wakeups only when ->deferred_qs is false, that is, on the first rcu_read_unlock() requiring special attention. Note that a chain of RCU readers linked by some other sort of reader may find that a later rcu_read_unlock() is once again able to do a full wakeup, courtesy of an intervening preemption:

Quoted 11 lines
	rcu_read_lock();
	/* preempted */
	local_irq_disable();
	rcu_read_unlock(); /* Can do full wakeup, sets ->deferred_qs. */
	rcu_read_lock();
	local_irq_enable();
	preempt_disable()
	rcu_read_unlock(); /* Cannot do full wakeup, ->deferred_qs set. */
	rcu_read_lock();
	preempt_enable();
	/* preempted, >deferred_qs reset. */

As it would have without ->deferred_sq and just having done the above which was deemed insufficient.

So I'm really puzzled by the need for all this.

Quoted 61 lines
	local_irq_disable();
	rcu_read_unlock(); /* Can again do full wakeup, sets ->deferred_qs. */

Such linked RCU readers do not yet seem to appear in the Linux kernel, and it is probably best if they don't. However, RCU needs to handle them, and some variations on this theme could make even raise_softirq() unsafe due to the possibility of its doing a full wakeup. This commit therefore also avoids invoking raise_softirq() when the ->deferred_qs set flag is set.

Cc Sebastian Andrzej Siewior <[email protected]>
---
 include/linux/sched.h    |  2 +-
 kernel/rcu/tree_plugin.h | 19 ++++++++++++++-----
 2 files changed, 15 insertions(+), 6 deletions(-)
include/linux/sched.h +1 -1
index 1549584a1538..3164b6798fe5 100644
@@ -566,7 +566,7 @@ union rcu_special {
 		u8			blocked;
 		u8			need_qs;
 		u8			exp_hint; /* Hint for performance. */
-		u8			pad; /* No garbage from compiler! */
+		u8			deferred_qs;
 	} b; /* Bits. */
 	u32 s; /* Set of bits. */
 };
kernel/rcu/tree_plugin.h +12 -4
index 21611862e083..75110ea75d01 100644
@@ -455,6 +455,7 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
 		local_irq_restore(flags);
 		return;
 	}
+	t->rcu_read_unlock_special.b.deferred_qs = false;
 	if (special.b.need_qs) {
 		rcu_qs();
 		t->rcu_read_unlock_special.b.need_qs = false;
@@ -605,16 +606,24 @@ static void rcu_read_unlock_special(struct task_struct *t)
 	local_irq_save(flags);
 	irqs_were_disabled = irqs_disabled_flags(flags);
 	if (preempt_bh_were_disabled || irqs_were_disabled) {
-		WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, false);
-		/* Need to defer quiescent state until everything is enabled. */
-		if (irqs_were_disabled && use_softirq) {
-			/* Enabling irqs does not reschedule, so... */
+		t->rcu_read_unlock_special.b.exp_hint = false;
+		// Need to defer quiescent state until everything is enabled.
+		if (irqs_were_disabled && use_softirq &&
+		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
+			// Using softirq, safe to awaken, and we get
+			// no help from enabling irqs, unlike bh/preempt.
 			raise_softirq_irqoff(RCU_SOFTIRQ);
+		} else if (irqs_were_disabled && !use_softirq &&
+			   !t->rcu_read_unlock_special.b.deferred_qs) {
+			// Safe to awaken and we get no help from enabling
+			// irqs, unlike bh/preempt.
+			invoke_rcu_core();

kernel/rcu/tree_plugin.h static void rcu_read_unlock_special(struct task_struct *t) · line 629

Quoted 10 lines
kernel/rcu/tree_plugin.h +2 -1
 		} else {
-			/* Enabling BH or preempt does reschedule, so... */
+			// Enabling BH or preempt does reschedule, so...
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
 		}
+		t->rcu_read_unlock_special.b.deferred_qs = true;
 		local_irq_restore(flags);
 		return;
 	}

That is all quite horrible and begging for sane solution. Would not something like:

static void rcu_force_resched(struct irq_work *work)

{
	set_tsk_need_resched(current);
	set_preempt_need_resched();
}
	if (irqs_were_disabled)
		irq_work_queue(&t->irq_work, rcu_force_resched);

Solve the whole thing?

#4 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Paul E. McKenney

When RCU core processing is offloaded from RCU_SOFTIRQ to the rcuc kthreads, a full and unconditional wakeup is required to initiate RCU core processing. In contrast, when RCU core processing is carried out by RCU_SOFTIRQ, a raise_softirq() suffices. Of course, there are situations where raise_softirq() does a full wakeup, but these do not occur with normal usage of rcu_read_unlock().

Do we have a comment somewhere explaining why?

First, thank you for reviewing this!

The "why" is because people normally don't do things like the code sequence shown below, but where the scheduler holds locks across the second RCU read-side critical section. (If they did, lockdep would complain. Nevertheless, it is good to avoid this potential problem.)

Quoted 9 lines
Quoted 6 lines

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This is all PREEMPT=y kernels, right? Where is the latency coming from? Because PREEMPT=y _should_ react quite quickly.

Yes, PREEMPT=y. It happens like this:

1. rcu_read_lock() with everything enabled.

2. Preemption then resumption.

3. local_irq_disable().

4. rcu_read_unlock().

5. local_irq_enable().

From what I know, the scheduler doesn't see anything until the next interrupt, local_bh_enable(), return to userspace, etc. Because this is PREEMPT=y, preempt_enable() and cond_resched() do nothing. So it could be some time (milliseconds, depending on HZ, NO_HZ_FULL, and so on) until the scheduler responds. With NO_HZ_FULL, last I knew, the delay can be extremely long.

Or am I missing something that gets the scheduler on the job faster?

Hmmm... If your point is that this amount of delay matters only for expedited grace periods, you are quite right. So perhaps I shouldn't be doing any of the expensive stuff unless there is an expedited grace period in flight. Or if NO_HZ_FULL. See below for an updated (and untested) patch to this effect.

Quoted 22 lines
Quoted 10 lines

This commit therefore introduces a ->deferred_qs flag to the task_struct structure's rcu_special structure. This flag is initially false, and is set to true by the first call to rcu_read_unlock() requiring special attention, then finally reset back to false when the quiescent state is finally reported. Then rcu_read_unlock() attempts full wakeups only when ->deferred_qs is false, that is, on the first rcu_read_unlock() requiring special attention. Note that a chain of RCU readers linked by some other sort of reader may find that a later rcu_read_unlock() is once again able to do a full wakeup, courtesy of an intervening preemption:

Quoted 11 lines
	rcu_read_lock();
	/* preempted */
	local_irq_disable();
	rcu_read_unlock(); /* Can do full wakeup, sets ->deferred_qs. */
	rcu_read_lock();
	local_irq_enable();
	preempt_disable()
	rcu_read_unlock(); /* Cannot do full wakeup, ->deferred_qs set. */
	rcu_read_lock();
	preempt_enable();
	/* preempted, >deferred_qs reset. */
Quoted 4 lines

As it would have without ->deferred_sq and just having done the above which was deemed insufficient.

So I'm really puzzled by the need for all this.

On the first round, without the ->deferred_qs, we know the scheduler cannot be holding any of its pi or rq locks because if it did, it would have disabled interrupts across the entire RCU read-side critical section. Wakeups are therefore safe in this case, whether via softirq or wakeup. Afterwards, we don't have that guarantee because an earlier critical section might have been preempted and the scheduler might have held one of its locks across the entire just-ended critical section.

And I believe you are right that we should avoid the wakeups unless there is an expedited grace period in flight or in a NO_HZ_FULL kernel. Hence the patch shown below.

Thanx, Paul

-----------------------------------------------------------------------

kernel/rcu/tree_plugin.h +8 -2
index 2e52a77af6be..582c6d88aaa0 100644
@@ -606,20 +606,26 @@ static void rcu_read_unlock_special(struct task_struct *t)
 	local_irq_save(flags);
 	irqs_were_disabled = irqs_disabled_flags(flags);
 	if (preempt_bh_were_disabled || irqs_were_disabled) {
+		bool exp;
+
 		t->rcu_read_unlock_special.b.exp_hint = false;
+		exp = !!READ_ONCE(this_cpu_ptr(&rcu_data)->mynode->exp_tasks);
 		// Need to defer quiescent state until everything is enabled.
-		if (irqs_were_disabled && use_softirq &&
+		if ((exp || IS_ENABLED(CONFIG_NO_HZ_FULL)) &&
+		    irqs_were_disabled && use_softirq &&
 		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
 			// Using softirq, safe to awaken, and we get
 			// no help from enabling irqs, unlike bh/preempt.
 			raise_softirq_irqoff(RCU_SOFTIRQ);
-		} else if (irqs_were_disabled && !use_softirq &&
+		} else if ((exp || IS_ENABLED(CONFIG_NO_HZ_FULL)) &&
+			   irqs_were_disabled && !use_softirq &&
 			   !t->rcu_read_unlock_special.b.deferred_qs) {
 			// Safe to awaken and we get no help from enabling
 			// irqs, unlike bh/preempt.
 			invoke_rcu_core();
 		} else {
 			// Enabling BH or preempt does reschedule, so...
+			// Also if no expediting or NO_HZ_FULL, slow is OK.
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
 		}
#5 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 90 lines · Paul E. McKenney
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Paul E. McKenney

When RCU core processing is offloaded from RCU_SOFTIRQ to the rcuc kthreads, a full and unconditional wakeup is required to initiate RCU core processing. In contrast, when RCU core processing is carried out by RCU_SOFTIRQ, a raise_softirq() suffices. Of course, there are situations where raise_softirq() does a full wakeup, but these do not occur with normal usage of rcu_read_unlock().

Do we have a comment somewhere explaining why?

First, thank you for reviewing this!

The "why" is because people normally don't do things like the code sequence shown below, but where the scheduler holds locks across the second RCU read-side critical section. (If they did, lockdep would complain. Nevertheless, it is good to avoid this potential problem.)

Quoted 9 lines
Quoted 6 lines

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This is all PREEMPT=y kernels, right? Where is the latency coming from? Because PREEMPT=y _should_ react quite quickly.

Yes, PREEMPT=y. It happens like this:

1. rcu_read_lock() with everything enabled.

2. Preemption then resumption.

3. local_irq_disable().

4. rcu_read_unlock().

5. local_irq_enable().

From what I know, the scheduler doesn't see anything until the next interrupt, local_bh_enable(), return to userspace, etc. Because this is PREEMPT=y, preempt_enable() and cond_resched() do nothing. So it could be some time (milliseconds, depending on HZ, NO_HZ_FULL, and so on) until the scheduler responds. With NO_HZ_FULL, last I knew, the delay can be extremely long.

Or am I missing something that gets the scheduler on the job faster?

Hmmm... If your point is that this amount of delay matters only for expedited grace periods, you are quite right. So perhaps I shouldn't be doing any of the expensive stuff unless there is an expedited grace period in flight. Or if NO_HZ_FULL. See below for an updated (and untested) patch to this effect.

Quoted 22 lines
Quoted 10 lines

This commit therefore introduces a ->deferred_qs flag to the task_struct structure's rcu_special structure. This flag is initially false, and is set to true by the first call to rcu_read_unlock() requiring special attention, then finally reset back to false when the quiescent state is finally reported. Then rcu_read_unlock() attempts full wakeups only when ->deferred_qs is false, that is, on the first rcu_read_unlock() requiring special attention. Note that a chain of RCU readers linked by some other sort of reader may find that a later rcu_read_unlock() is once again able to do a full wakeup, courtesy of an intervening preemption:

Quoted 11 lines
	rcu_read_lock();
	/* preempted */
	local_irq_disable();
	rcu_read_unlock(); /* Can do full wakeup, sets ->deferred_qs. */
	rcu_read_lock();
	local_irq_enable();
	preempt_disable()
	rcu_read_unlock(); /* Cannot do full wakeup, ->deferred_qs set. */
	rcu_read_lock();
	preempt_enable();
	/* preempted, >deferred_qs reset. */
Quoted 4 lines

As it would have without ->deferred_sq and just having done the above which was deemed insufficient.

So I'm really puzzled by the need for all this.

On the first round, without the ->deferred_qs, we know the scheduler cannot be holding any of its pi or rq locks because if it did, it would have disabled interrupts across the entire RCU read-side critical section. Wakeups are therefore safe in this case, whether via softirq or wakeup. Afterwards, we don't have that guarantee because an earlier critical section might have been preempted and the scheduler might have held one of its locks across the entire just-ended critical section.

Quoted 3 lines

And I believe you are right that we should avoid the wakeups unless there is an expedited grace period in flight or in a NO_HZ_FULL kernel. Hence the patch shown below.

Which is a stupid patch. It assumes ancient times when expedited grace periods were guaranteed to preempted pre-existing RCU read-side critical sections. Back to the drawing board...

Thanx, Paul

Quoted 35 lines

-----------------------------------------------------------------------

kernel/rcu/tree_plugin.h +8 -2
index 2e52a77af6be..582c6d88aaa0 100644
@@ -606,20 +606,26 @@ static void rcu_read_unlock_special(struct task_struct *t)
 	local_irq_save(flags);
 	irqs_were_disabled = irqs_disabled_flags(flags);
 	if (preempt_bh_were_disabled || irqs_were_disabled) {
+		bool exp;
+
 		t->rcu_read_unlock_special.b.exp_hint = false;
+		exp = !!READ_ONCE(this_cpu_ptr(&rcu_data)->mynode->exp_tasks);
 		// Need to defer quiescent state until everything is enabled.
-		if (irqs_were_disabled && use_softirq &&
+		if ((exp || IS_ENABLED(CONFIG_NO_HZ_FULL)) &&
+		    irqs_were_disabled && use_softirq &&
 		    (in_irq() || !t->rcu_read_unlock_special.b.deferred_qs)) {
 			// Using softirq, safe to awaken, and we get
 			// no help from enabling irqs, unlike bh/preempt.
 			raise_softirq_irqoff(RCU_SOFTIRQ);
-		} else if (irqs_were_disabled && !use_softirq &&
+		} else if ((exp || IS_ENABLED(CONFIG_NO_HZ_FULL)) &&
+			   irqs_were_disabled && !use_softirq &&
 			   !t->rcu_read_unlock_special.b.deferred_qs) {
 			// Safe to awaken and we get no help from enabling
 			// irqs, unlike bh/preempt.
 			invoke_rcu_core();
 		} else {
 			// Enabling BH or preempt does reschedule, so...
+			// Also if no expediting or NO_HZ_FULL, slow is OK.
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
 		}
maintainer
#6 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

In reply to Paul E. McKenney
Quoted 22 lines · Paul E. McKenney
Quoted 9 lines
Quoted 6 lines

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This is all PREEMPT=y kernels, right? Where is the latency coming from? Because PREEMPT=y _should_ react quite quickly.

Yes, PREEMPT=y. It happens like this:

1. rcu_read_lock() with everything enabled.

2. Preemption then resumption.

3. local_irq_disable().

4. rcu_read_unlock().

5. local_irq_enable().

Quoted 8 lines

From what I know, the scheduler doesn't see anything until the next interrupt, local_bh_enable(), return to userspace, etc. Because this is PREEMPT=y, preempt_enable() and cond_resched() do nothing. So it could be some time (milliseconds, depending on HZ, NO_HZ_FULL, and so on) until the scheduler responds. With NO_HZ_FULL, last I knew, the delay can be extremely long.

Or am I missing something that gets the scheduler on the job faster?

Oh urgh, yah. So normally we only twiddle with the need_resched state:

- while preempt_disabl(), such that preempt_enable() will reschedule
- from interrupt context, such that interrupt return will reschedule

But the usage here 'violates' those rules and then there is an unspecified latency between setting the state and it getting observed, but no longer than 1 tick I would think.

I don't think we can go NOHZ with need_resched set, because the moment we hit the idle loop with that set, we _will_ reschedule.

So in that respect the irq_work suggestion I made would fix things properly.

#7 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 32 lines · Peter Zijlstra
Quoted 22 lines · Paul E. McKenney
Quoted 9 lines
Quoted 6 lines

The initial solution to this problem was to use set_tsk_need_resched() and set_preempt_need_resched() to force a future context switch, which allows rcu_preempt_note_context_switch() to report the deferred quiescent state to RCU's core processing. Unfortunately for expedited grace periods, there can be a significant delay between the call for a context switch and the actual context switch.

This is all PREEMPT=y kernels, right? Where is the latency coming from? Because PREEMPT=y _should_ react quite quickly.

Yes, PREEMPT=y. It happens like this:

1. rcu_read_lock() with everything enabled.

2. Preemption then resumption.

3. local_irq_disable().

4. rcu_read_unlock().

5. local_irq_enable().

Quoted 8 lines

From what I know, the scheduler doesn't see anything until the next interrupt, local_bh_enable(), return to userspace, etc. Because this is PREEMPT=y, preempt_enable() and cond_resched() do nothing. So it could be some time (milliseconds, depending on HZ, NO_HZ_FULL, and so on) until the scheduler responds. With NO_HZ_FULL, last I knew, the delay can be extremely long.

Or am I missing something that gets the scheduler on the job faster?

Quoted 8 lines

Oh urgh, yah. So normally we only twiddle with the need_resched state:

- while preempt_disabl(), such that preempt_enable() will reschedule
- from interrupt context, such that interrupt return will reschedule

But the usage here 'violates' those rules and then there is an unspecified latency between setting the state and it getting observed, but no longer than 1 tick I would think.

In general, yes, which is fine (famous last words) for normal grace periods but not so good for expedited grace periods.

Quoted 2 lines

I don't think we can go NOHZ with need_resched set, because the moment we hit the idle loop with that set, we _will_ reschedule.

Agreed, and I believe that transitioning to usermode execution also gives the scheduler a chance to take action.

The one exception to this is when a nohz_full CPU running in nohz_full mode does a system call that decides to execute for a very long time. Last I checked, the scheduling-clock interrupt did -not- get retriggered in this case, and the delay could be indefinite, as in bad even for normal grace periods.

Quoted 2 lines

So in that respect the irq_work suggestion I made would fix things properly.

But wouldn't the current use of set_tsk_need_resched(current) followed by set_preempt_need_resched() work just as well in that case? The scheduler would react to these at the next scheduler-clock interrupt on their own, right? Or am I being scheduler-naive again?

Thanx, Paul

maintainer
#8 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 20 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney

Or am I missing something that gets the scheduler on the job faster?

Quoted 8 lines

Oh urgh, yah. So normally we only twiddle with the need_resched state:

- while preempt_disabl(), such that preempt_enable() will reschedule
- from interrupt context, such that interrupt return will reschedule

But the usage here 'violates' those rules and then there is an unspecified latency between setting the state and it getting observed, but no longer than 1 tick I would think.

In general, yes, which is fine (famous last words) for normal grace periods but not so good for expedited grace periods.

Quoted 2 lines

I don't think we can go NOHZ with need_resched set, because the moment we hit the idle loop with that set, we _will_ reschedule.

Quoted 8 lines

Agreed, and I believe that transitioning to usermode execution also gives the scheduler a chance to take action.

The one exception to this is when a nohz_full CPU running in nohz_full mode does a system call that decides to execute for a very long time. Last I checked, the scheduling-clock interrupt did -not- get retriggered in this case, and the delay could be indefinite, as in bad even for normal grace periods.

Right, there is that.

Quoted 7 lines
Quoted 2 lines

So in that respect the irq_work suggestion I made would fix things properly.

But wouldn't the current use of set_tsk_need_resched(current) followed by set_preempt_need_resched() work just as well in that case? The scheduler would react to these at the next scheduler-clock interrupt on their own, right? Or am I being scheduler-naive again?

Well, you have that unspecified delay. By forcing the (self) interrupt you enforce a timely response.

#9 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 35 lines · Peter Zijlstra
Quoted 20 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney

Or am I missing something that gets the scheduler on the job faster?

Quoted 8 lines

Oh urgh, yah. So normally we only twiddle with the need_resched state:

- while preempt_disabl(), such that preempt_enable() will reschedule
- from interrupt context, such that interrupt return will reschedule

But the usage here 'violates' those rules and then there is an unspecified latency between setting the state and it getting observed, but no longer than 1 tick I would think.

In general, yes, which is fine (famous last words) for normal grace periods but not so good for expedited grace periods.

Quoted 2 lines

I don't think we can go NOHZ with need_resched set, because the moment we hit the idle loop with that set, we _will_ reschedule.

Quoted 8 lines

Agreed, and I believe that transitioning to usermode execution also gives the scheduler a chance to take action.

The one exception to this is when a nohz_full CPU running in nohz_full mode does a system call that decides to execute for a very long time. Last I checked, the scheduling-clock interrupt did -not- get retriggered in this case, and the delay could be indefinite, as in bad even for normal grace periods.

Right, there is that.

Quoted 3 lines
Quoted 2 lines

So in that respect the irq_work suggestion I made would fix things properly.

Quoted 7 lines
Quoted 4 lines

But wouldn't the current use of set_tsk_need_resched(current) followed by set_preempt_need_resched() work just as well in that case? The scheduler would react to these at the next scheduler-clock interrupt on their own, right? Or am I being scheduler-naive again?

Well, you have that unspecified delay. By forcing the (self) interrupt you enforce a timely response.

Good point! I will give this a go, thank you!

Thanx, Paul

#10 lore raw

Re: [PATCH tip/core/rcu 2/2] rcu: Check for wakeup-safe conditions in rcu_read_unlock_special()

Quoted 41 lines · Paul E. McKenney
Quoted 40 lines · Peter Zijlstra
Quoted 20 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Quoted 1 line · Paul E. McKenney

Or am I missing something that gets the scheduler on the job faster?

Quoted 8 lines

Oh urgh, yah. So normally we only twiddle with the need_resched state:

- while preempt_disabl(), such that preempt_enable() will reschedule
- from interrupt context, such that interrupt return will reschedule

But the usage here 'violates' those rules and then there is an unspecified latency between setting the state and it getting observed, but no longer than 1 tick I would think.

In general, yes, which is fine (famous last words) for normal grace periods but not so good for expedited grace periods.

Quoted 2 lines

I don't think we can go NOHZ with need_resched set, because the moment we hit the idle loop with that set, we _will_ reschedule.

Quoted 8 lines

Agreed, and I believe that transitioning to usermode execution also gives the scheduler a chance to take action.

The one exception to this is when a nohz_full CPU running in nohz_full mode does a system call that decides to execute for a very long time. Last I checked, the scheduling-clock interrupt did -not- get retriggered in this case, and the delay could be indefinite, as in bad even for normal grace periods.

Right, there is that.

Quoted 7 lines
Quoted 2 lines

So in that respect the irq_work suggestion I made would fix things properly.

But wouldn't the current use of set_tsk_need_resched(current) followed by set_preempt_need_resched() work just as well in that case? The scheduler would react to these at the next scheduler-clock interrupt on their own, right? Or am I being scheduler-naive again?

Quoted 4 lines
Quoted 2 lines

Well, you have that unspecified delay. By forcing the (self) interrupt you enforce a timely response.

Good point! I will give this a go, thank you!

How about as shown below?

Thanx, Paul

------------------------------------------------------------------------

commit 687c00c91c9edbaf5309402689bce644dd140590

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 4 12:19:25 2019 -0700

rcu: Use irq_work to get scheduler's attention in clean context

When rcu_read_unlock_special() is invoked with interrupts disabled, is either not in an interrupt handler or is not using RCU_SOFTIRQ, is not the first RCU read-side critical section in the chain, and either there is an expedited grace period in flight or this is a NO_HZ_FULL kernel, the end of the grace period can be unduly delayed. The reason for this is that it is not safe to do wakeups in this situation.

This commit fixes this problem by using the irq_work subsystem to force a later interrupt handler in a clean environment. Because set_tsk_need_resched(current) and set_preempt_need_resched() are invoked prior to this, the scheduler will force a context switch upon return from this interrupt (though perhaps at the end of any interrupted preempt-disable or BH-disable region of code), which will invoke rcu_note_context_switch() (again in a clean environment), which will in turn give RCU the chance to report the deferred quiescent state.

Of course, by then this task might be within another RCU read-side critical section. But that will be detected at that time and reporting will be further deferred to the outermost rcu_read_unlock(). See rcu_preempt_need_deferred_qs() and rcu_preempt_deferred_qs() for more details on the checking.

kernel/rcu/tree.h +2 -0
index b9c5d1af8451..dc3c53cb9608 100644
@@ -161,6 +161,8 @@ struct rcu_data {
 					/*  ticks this CPU has handled */
 					/*  during and after the last grace */
 					/* period it is aware of. */
+	struct irq_work defer_qs_iw;	/* Obtain later scheduler attention. */
+	bool defer_qs_iw_pending;	/* Scheduler attention pending? */
 
 	/* 2) batch handling */
 	struct rcu_segcblist cblist;	/* Segmented callback list, with */
kernel/rcu/tree_plugin.h +20 -0
index d90a262ba04b..80ee4d3f3891 100644
@@ -587,6 +587,17 @@ static void rcu_preempt_deferred_qs(struct task_struct *t)
 		t->rcu_read_lock_nesting += RCU_NEST_BIAS;
 }
 
+/*
+ * Minimal handler to give the scheduler a chance to re-evaluate.
+ */
+static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
+{
+	struct rcu_data *rdp;
+
+	rdp = container_of(iwp, struct rcu_data, defer_qs_iw);
+	rdp->defer_qs_iw_pending = false;
+}
+
 /*
  * Handle special cases during rcu_read_unlock(), such as needing to
  * notify RCU core processing or task having blocked during the RCU
@@ -630,6 +641,15 @@ static void rcu_read_unlock_special(struct task_struct *t)
 			// Also if no expediting or NO_HZ_FULL, slow is OK.
 			set_tsk_need_resched(current);
 			set_preempt_need_resched();
+			if (IS_ENABLED(CONFIG_IRQ_WORK) &&
+			    !rdp->defer_qs_iw_pending && exp) {
+				// Get scheduler to re-evaluate and call hooks.
+				// If !IRQ_WORK, FQS scan will eventually IPI.
+				init_irq_work(&rdp->defer_qs_iw,
+					      rcu_preempt_deferred_qs_handler);
+				rdp->defer_qs_iw_pending = true;
+				irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
+			}
 		}
 		t->rcu_read_unlock_special.b.deferred_qs = true;
 		local_irq_restore(flags);