Skip to contentSkip to conversation

Miscellaneous fixes for v5.2

PATCH New Paul E. McKenney 1 participant 19 messages Mar 26, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Miscellaneous fixes for v5.2

Hello!

This series contains miscellaneous fixes.

1. Unconditionally expedite during suspend/hibernate (unless the
real-time guys have disabled expediting altogether, that is).

2. Avoid unnecessary softirq when system is idle, courtesy of
Joel Fernandes.

3. rcu_qs -- Use raise_softirq_irqoff to not save irqs twice,
courtesy of Cyrill Gorcunov.

4. Make exit_rcu() handle non-preempted RCU readers.

5. Set rcutree.kthread_prio sysfs access to read-only, courtesy
of Liu Song.

6. MAINTAINERS: RCU now has its own email list.

7. MAINTAINERS: Add -rcu branch name ("dev").

8. rcu: Move common code out of if-else block, courtesy of Akira
Yokosawa.

9. Allow rcu_nocbs= to specify all CPUs.

10. Report error for bad rcu_nocbs= parameter values.

11. Fix self-wakeups for grace-period kthread, courtesy of Neeraj
Upadhyay.

12. Default jiffies_to_sched_qs to jiffies_till_sched_qs, courtesy
of Neeraj Upadhyay.

13. Do a single rhp->func read in rcu_head_after_call_rcu(),
courtesy of Neeraj Upadhyay.

14. Update jiffies_to_sched_qs and adjust_jiffies_till_sched_qs()
comments.

15. Fix force_qs_rnp() header comment, courtesy of Zhouyi Zhou.

16. Eliminate redundant NULL-pointer check.

17. Fix typo in tree_exp.h comment.

18. Correct READ_ONCE()/WRITE_ONCE() for ->rcu_read_unlock_special.

Thanx, Paul

------------------------------------------------------------------------

 Documentation/admin-guide/kernel-parameters.txt |    4 +-
 MAINTAINERS                                     |   16 ++++----
 include/linux/rcupdate.h                        |    6 ++-
 kernel/rcu/tiny.c                               |    2 -
 kernel/rcu/tree.c                               |   31 +++++++--------
 kernel/rcu/tree_exp.h                           |    4 +-
 kernel/rcu/tree_plugin.h                        |   48 +++++++++++++++---------
 7 files changed, 63 insertions(+), 48 deletions(-)
#1 lore raw

PATCH 1/18 rcu: Unconditionally expedite during suspend/hibernate

The rcu_pm_notify() function refuses to switch to/from expedited grace periods on systems with more than 256 CPUs due to the serialized initialization of expedited grace periods. However, expedited grace periods are now initialized in parallel, removing this concern. This commit therefore removes the checks from rcu_pm_notify(), so that expedited grace periods are used unconditionally during suspend/resume and hibernate/wake operations.

As always, real-time workloads wishing to completely avoid expedited grace periods can use the rcupdate.rcu_normal= kernel parameter.

---
 kernel/rcu/tree.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +2 -4
index acd6ccf56faf..95e3250b7b6e 100644
@@ -3559,13 +3559,11 @@ static int rcu_pm_notify(struct notifier_block *self,
 	switch (action) {
 	case PM_HIBERNATION_PREPARE:
 	case PM_SUSPEND_PREPARE:
-		if (nr_cpu_ids <= 256) /* Expediting bad for large systems. */
-			rcu_expedite_gp();
+		rcu_expedite_gp();
 		break;
 	case PM_POST_HIBERNATION:
 	case PM_POST_SUSPEND:
-		if (nr_cpu_ids <= 256) /* Expediting bad for large systems. */
-			rcu_unexpedite_gp();
+		rcu_unexpedite_gp();
 		break;
 	default:
 		break;
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/18 rcu: Avoid unnecessary softirq when system is idle

In reply to Paul E. McKenney

From: "Joel Fernandes (Google)" <[email protected]>

When there are no callbacks pending on an idle system, I noticed that RCU softirq is continuously firing. During this the cpu_no_qs is set to false, and core_needs_qs is set to true indefinitely. This causes rcu_process_callbacks to be repeatedly called, even though the node corresponding to the CPU has that CPU's mask bit cleared and the system is idle. I believe the race is when such mask clearing is done during idle CPU scan of the quiescent state forcing stage in the kthread instead of the softirq. Since the rnp mask is cleared, but the flags on the CPU's rdp are not cleared, the CPU thinks it still needs to report to core RCU.

Cure this by clearing the core_needs_qs flag when the CPU detects that its node is already updated which will avoid the unwanted softirq raises to the benefit of real-time systems.

Test: Ran rcutorture for various tree RCU configs.

---
 kernel/rcu/tree.c | 1 +
 1 file changed, 1 insertion(+)
kernel/rcu/tree.c +1 -0
index 95e3250b7b6e..2f78a115d34c 100644
@@ -2296,6 +2296,7 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 	}
 	mask = rdp->grpmask;
 	if ((rnp->qsmask & mask) == 0) {
+		rdp->core_needs_qs = false;
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	} else {
 		rdp->core_needs_qs = false;
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/18 rcu: rcu_qs -- Use raise_softirq_irqoff to not save irqs twice

In reply to Paul E. McKenney

From: Cyrill Gorcunov <[email protected]>

The rcu_qs is disabling IRQs by self so no need to do the same in raise_softirq but instead we can save some cycles using raise_softirq_irqoff directly.

---
 kernel/rcu/tiny.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tiny.c +1 -1
index 911bd9076d43..477b4eb44af5 100644
@@ -52,7 +52,7 @@ void rcu_qs(void)
 	local_irq_save(flags);
 	if (rcu_ctrlblk.donetail != rcu_ctrlblk.curtail) {
 		rcu_ctrlblk.donetail = rcu_ctrlblk.curtail;
-		raise_softirq(RCU_SOFTIRQ);
+		raise_softirq_irqoff(RCU_SOFTIRQ);
 	}
 	local_irq_restore(flags);
 }
Signature · 1 line
2.17.1
#4 lore raw

PATCH 4/18 rcu: Make exit_rcu() handle non-preempted RCU readers

In reply to Paul E. McKenney

The purpose of exit_rcu() is to handle cases where buggy code causes a task to exit within an RCU read-side critical section. It currently does that in the case where said RCU read-side critical section was preempted at least once, but fails to handle cases where preemption did not occur. This case needs to be handled because otherwise the final context switch away from the exiting task will incorrectly behave as if task exit were instead a preemption of an RCU read-side critical section, and will therefore queue the exiting task. The exiting task will have exited, and thus won't ever execute rcu_read_unlock(), which means that it will remain queued forever, blocking all subsequent grace periods, and eventually resulting in OOM.

Although this is arguably better than letting grace periods proceed and having a later rcu_read_unlock() access the now-freed task structure that once belonged to the exiting tasks, it would obviously be better to correctly handle this case. This commit therefore sets ->rcu_read_lock_nesting to 1 in that case, so that the subsequence call to __rcu_read_unlock() causes the exiting task to exit its dangling RCU read-side critical section.

Note that deferred quiescent states need not be considered. The reason is that removing the task from the ->blkd_tasks[] list in the call to rcu_preempt_deferred_qs() handles the per-task component of any deferred quiescent state, and all other components of any deferred quiescent state are associated with the CPU, which isn't going anywhere until some later CPU-hotplug operation, which will report any remaining deferred quiescent states from within the rcu_report_dead() function.

Note also that negative values of ->rcu_read_lock_nesting need not be considered. First, these won't show up in exit_rcu() unless there is a serious bug in RCU, and second, setting ->rcu_read_lock_nesting sets the state so that the RCU read-side critical section will be exited normally.

Again, this code has no effect unless there has been some prior bug that prevents a task from leaving an RCU read-side critical section before exiting. Furthermore, there have been no reports of the bug fixed by this commit appearing in production. This commit is therefore absolutely -not- recommended for backporting to -stable.

---
 kernel/rcu/tree_plugin.h | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
kernel/rcu/tree_plugin.h +13 -7
index 97dba50f6fb2..d408661d5fb7 100644
@@ -804,19 +804,25 @@ static void rcu_flavor_sched_clock_irq(int user)
 
 /*
  * Check for a task exiting while in a preemptible-RCU read-side
- * critical section, clean up if so.  No need to issue warnings,
- * as debug_check_no_locks_held() already does this if lockdep
- * is enabled.
+ * critical section, clean up if so.  No need to issue warnings, as
+ * debug_check_no_locks_held() already does this if lockdep is enabled.
+ * Besides, if this function does anything other than just immediately
+ * return, there was a bug of some sort.  Spewing warnings from this
+ * function is like as not to simply obscure important prior warnings.
  */
 void exit_rcu(void)
 {
 	struct task_struct *t = current;
 
-	if (likely(list_empty(&current->rcu_node_entry)))
+	if (unlikely(!list_empty(&current->rcu_node_entry))) {
+		t->rcu_read_lock_nesting = 1;
+		barrier();
+		t->rcu_read_unlock_special.b.blocked = true;
+	} else if (unlikely(t->rcu_read_lock_nesting)) {
+		t->rcu_read_lock_nesting = 1;
+	} else {
 		return;
-	t->rcu_read_lock_nesting = 1;
-	barrier();
-	t->rcu_read_unlock_special.b.blocked = true;
+	}
 	__rcu_read_unlock();
 	rcu_preempt_deferred_qs(current);
 }
Signature · 1 line
2.17.1
#5 lore raw

PATCH 5/18 rcu: Set rcutree.kthread_prio sysfs access to read-only

In reply to Paul E. McKenney

From: Liu Song <[email protected]>

The rcutree.kthread_prio kernel-boot parameter is used to set the priority for boost (rcub), per-CPU (rcuc), and grace-period (rcu_preempt or rcu_sched) kthreads. It is also used by rcutorture to check whether it is possible to meaningfully test RCU priority boosting. However, all of these cases will either ignore or be confused by any post-boot changes to rcutree.kthread_prio.

Note that the user really can change the priorities of all of these kthreads using chrt, given sufficient privileges. Therefore, the read-write nature of sysfs access to rcutree.kthread_prio is thus at best an attractive nuisance.

This commit therefore changes sysfs access to rcutree.kthread_prio to be read-only.

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index 2f78a115d34c..296131450414 100644
@@ -149,7 +149,7 @@ static void sync_sched_exp_online_cleanup(int cpu);
 
 /* rcuc/rcub kthread realtime priority */
 static int kthread_prio = IS_ENABLED(CONFIG_RCU_BOOST) ? 1 : 0;
-module_param(kthread_prio, int, 0644);
+module_param(kthread_prio, int, 0444);
 
 /* Delay in jiffies for grace-period initialization delays, debug only. */
 
Signature · 1 line
2.17.1
#6 lore raw

PATCH 6/18 MAINTAINERS: RCU now has its own email list

In reply to Paul E. McKenney

This commit makes [email protected] be the official list for RCU-related topics.

---
 MAINTAINERS | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
MAINTAINERS +3 -3
index e17ebf70b548..1924b52937a6 100644
@@ -13031,7 +13031,7 @@ M:	Josh Triplett <[email protected]>
 R:	Steven Rostedt <[email protected]>
 R:	Mathieu Desnoyers <[email protected]>
 R:	Lai Jiangshan <[email protected]>
-L:	[email protected]
+L:	[email protected]
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
 F:	tools/testing/selftests/rcutorture
@@ -13079,7 +13079,7 @@ R:	Steven Rostedt <[email protected]>
 R:	Mathieu Desnoyers <[email protected]>
 R:	Lai Jiangshan <[email protected]>
 R:	Joel Fernandes <[email protected]>
-L:	[email protected]
+L:	[email protected]
 W:	http://www.rdrop.com/users/paulmck/RCU/
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
@@ -14234,7 +14234,7 @@ M:	"Paul E. McKenney" <[email protected]>
 M:	Josh Triplett <[email protected]>
 R:	Steven Rostedt <[email protected]>
 R:	Mathieu Desnoyers <[email protected]>
-L:	[email protected]
+L:	[email protected]
 W:	http://www.rdrop.com/users/paulmck/RCU/
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
Signature · 1 line
2.17.1
#7 lore raw

PATCH 7/18 MAINTAINERS: Add -rcu branch name ("dev")

In reply to Paul E. McKenney
---
 MAINTAINERS | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
MAINTAINERS +5 -5
index 1924b52937a6..a9b5270d006e 100644
@@ -8983,7 +8983,7 @@ R:	Daniel Lustig <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Supported
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
 F:	tools/memory-model/
 F:	Documentation/atomic_bitops.txt
 F:	Documentation/atomic_t.txt
@@ -13033,7 +13033,7 @@ R:	Mathieu Desnoyers <[email protected]>
 R:	Lai Jiangshan <[email protected]>
 L:	[email protected]
 S:	Supported
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
 F:	tools/testing/selftests/rcutorture
 
 RDC R-321X SoC
@@ -13082,7 +13082,7 @@ R:	Joel Fernandes <[email protected]>
 L:	[email protected]
 W:	http://www.rdrop.com/users/paulmck/RCU/
 S:	Supported
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
 F:	Documentation/RCU/
 X:	Documentation/RCU/torture.txt
 F:	include/linux/rcu*
@@ -14237,7 +14237,7 @@ R:	Mathieu Desnoyers <[email protected]>
 L:	[email protected]
 W:	http://www.rdrop.com/users/paulmck/RCU/
 S:	Supported
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
 F:	include/linux/srcu*.h
 F:	kernel/rcu/srcu*.c
 
@@ -15684,7 +15684,7 @@ M:	"Paul E. McKenney" <[email protected]>
 M:	Josh Triplett <[email protected]>
 L:	[email protected]
 S:	Supported
-T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev
 F:	Documentation/RCU/torture.txt
 F:	kernel/torture.c
 F:	kernel/rcu/rcutorture.c
Signature · 1 line
2.17.1
#8 lore raw

PATCH 8/18 rcu: Move common code out of if-else block

In reply to Paul E. McKenney

From: Akira Yokosawa <[email protected]>

As the result of recent addition of "rdp->core_needs_qs = false;" in the "if" block, now both branches of the if-else have the same assignment.

Factor it out and reduce line count.

---
 kernel/rcu/tree.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
kernel/rcu/tree.c +1 -3
index 296131450414..5aefd36ac648 100644
@@ -2295,12 +2295,10 @@ rcu_report_qs_rdp(int cpu, struct rcu_data *rdp)
 		return;
 	}
 	mask = rdp->grpmask;
+	rdp->core_needs_qs = false;
 	if ((rnp->qsmask & mask) == 0) {
-		rdp->core_needs_qs = false;
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	} else {
-		rdp->core_needs_qs = false;
-
 		/*
 		 * This GP can't end until cpu checks in, so all of our
 		 * callbacks can be processed during the next GP.
Signature · 1 line
2.17.1
#9 lore raw

PATCH 9/18 rcu: Allow rcu_nocbs= to specify all CPUs

In reply to Paul E. McKenney

Currently, the rcu_nocbs= kernel boot parameter requires that a specific list of CPUs be specified, and has no way to say "all of them". As noted by user RavFX in a comment to Phoronix topic 1002538, this is an inconvenient side effect of the removal of the RCU_NOCB_CPU_ALL Kconfig option. This commit therefore enables the rcu_nocbs= kernel boot parameter to be given the string "all", as in "rcu_nocbs=all" to specify that all CPUs on the system are to have their RCU callbacks offloaded.

Another approach would be to make cpulist_parse() check for "all", but there are uses of cpulist_parse() that do other checking, which could conflict with an "all". This commit therefore focuses on the specific use of cpulist_parse() in rcu_nocb_setup().

Just a note to other people who would like changes to Linux-kernel RCU:
If you send your requests to me directly, they might get fixed somewhat faster. RavFX's comment was posted on January 22, 2018 and I first saw it on March 5, 2019. And the only reason that I found it -at- -all- was that I was looking for projects using RCU, and my search engine showed me that Phoronix comment quite by accident. Your choice, though! ;-)

---
 Documentation/admin-guide/kernel-parameters.txt | 4 +++-
 kernel/rcu/tree_plugin.h                        | 5 ++++-
 2 files changed, 7 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -1
index 2b8ee90bb644..d377a2166b79 100644
@@ -3623,7 +3623,9 @@
 				see CONFIG_RAS_CEC help text.
 
 	rcu_nocbs=	[KNL]
-			The argument is a cpu list, as described above.
+			The argument is a cpu list, as described above,
+			except that the string "all" can be used to
+			specify every CPU on the system.
 
 			In kernels built with CONFIG_RCU_NOCB_CPU=y, set
 			the specified list of CPUs to be no-callback CPUs.
kernel/rcu/tree_plugin.h +4 -1
index d408661d5fb7..ed4a6dabf31d 100644
@@ -1776,7 +1776,10 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp)
 static int __init rcu_nocb_setup(char *str)
 {
 	alloc_bootmem_cpumask_var(&rcu_nocb_mask);
-	cpulist_parse(str, rcu_nocb_mask);
+	if (!strcasecmp(str, "all"))
+		cpumask_setall(rcu_nocb_mask);
+	else
+		cpulist_parse(str, rcu_nocb_mask);
 	return 1;
 }
 __setup("rcu_nocbs=", rcu_nocb_setup);
Signature · 1 line
2.17.1
#10 lore raw

PATCH 10/18 rcu: Report error for bad rcu_nocbs= parameter values

In reply to Paul E. McKenney

This commit prints a console message when cpulist_parse() reports a bad list of CPUs, and sets all CPUs' bits in that case. The reason for setting all CPUs' bits is that this is the safe(r) choice for real-time workloads, which would normally be the ones using the rcu_nocbs= kernel boot parameter. Either way, later RCU console log messages list the actual set of CPUs whose RCU callbacks will be offloaded.

---
 kernel/rcu/tree_plugin.h | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +10 -2
index ed4a6dabf31d..f0aeb7416dcc 100644
@@ -1772,14 +1772,22 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp)
  */
 
 
-/* Parse the boot-time rcu_nocb_mask CPU list from the kernel parameters. */
+/*
+ * Parse the boot-time rcu_nocb_mask CPU list from the kernel parameters.
+ * The string after the "rcu_nocbs=" is either "all" for all CPUs, or a
+ * comma-separated list of CPUs and/or CPU ranges.  If an invalid list is
+ * given, a warning is emitted and all CPUs are offloaded.
+ */
 static int __init rcu_nocb_setup(char *str)
 {
 	alloc_bootmem_cpumask_var(&rcu_nocb_mask);
 	if (!strcasecmp(str, "all"))
 		cpumask_setall(rcu_nocb_mask);
 	else
-		cpulist_parse(str, rcu_nocb_mask);
+		if (cpulist_parse(str, rcu_nocb_mask)) {
+			pr_warn("rcu_nocbs= bad CPU range, all CPUs set\n");
+			cpumask_setall(rcu_nocb_mask);
+		}
 	return 1;
 }
 __setup("rcu_nocbs=", rcu_nocb_setup);
Signature · 1 line
2.17.1
#11 lore raw

PATCH 11/18 rcu: Fix self-wakeups for grace-period kthread

In reply to Paul E. McKenney

From: Neeraj Upadhyay <[email protected]>

The current rcu_gp_kthread_wake() function uses in_interrupt() and thus does a self-wakeup from all interrupt contexts, including the pointless case where the GP kthread happens to be running with bottom halves disabled, along with the impossible case where the GP kthread is running within an NMI handler (you are not supposed to invoke rcu_gp_kthread_wake() from within an NMI handler. This commit therefore replaces the in_interrupt() with in_irq(), so that the self-wakeups happen only from handlers for hardware interrupts and softirqs. This also makes the code match the comment.

---
 kernel/rcu/tree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
index 5aefd36ac648..139fa1f5c537 100644
@@ -1585,7 +1585,7 @@ static bool rcu_future_gp_cleanup(struct rcu_node *rnp)
 static void rcu_gp_kthread_wake(void)
 {
 	if ((current == rcu_state.gp_kthread &&
-	     !in_interrupt() && !in_serving_softirq()) ||
+	     !in_irq() && !in_serving_softirq()) ||
 	    !READ_ONCE(rcu_state.gp_flags) ||
 	    !rcu_state.gp_kthread)
 		return;
Signature · 1 line
2.17.1
#12 lore raw

PATCH 12/18 rcu: Default jiffies_to_sched_qs to jiffies_till_sched_qs

In reply to Paul E. McKenney

From: Neeraj Upadhyay <[email protected]>

The current code only calls adjust_jiffies_till_sched_qs() if jiffies_till_sched_qs is left at its default value, so when the jiffies_till_sched_qs kernel-boot parameter actually is specified, jiffies_to_sched_qs will be left with the value zero, which will result in useless slowdowns of cond_resched(). This commit therefore changes rcu_init_geometry() to unconditionally invoke adjust_jiffies_till_sched_qs(), which ensures that jiffies_to_sched_qs will be initialized in all cases, thus maintaining good cond_resched() performance.

---
 kernel/rcu/tree.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/tree.c +1 -2
index 139fa1f5c537..466299c3d2da 100644
@@ -3739,8 +3739,7 @@ static void __init rcu_init_geometry(void)
 		jiffies_till_first_fqs = d;
 	if (jiffies_till_next_fqs == ULONG_MAX)
 		jiffies_till_next_fqs = d;
-	if (jiffies_till_sched_qs == ULONG_MAX)
-		adjust_jiffies_till_sched_qs();
+	adjust_jiffies_till_sched_qs();
 
 	/* If the compile-time values are accurate, just leave. */
 	if (rcu_fanout_leaf == RCU_FANOUT_LEAF &&
Signature · 1 line
2.17.1
#13 lore raw

PATCH 13/18 rcu: Do a single rhp->func read in rcu_head_after_call_rcu()

In reply to Paul E. McKenney

From: Neeraj Upadhyay <[email protected]>

The rcu_head_after_call_rcu() function reads the rhp->func pointer twice, which can result in a false-positive WARN_ON_ONCE() if the callback were passed to call_rcu() between the two reads. Although racing rcu_head_after_call_rcu() with call_rcu() is to be a dubious use case (the return value is not reliable in that case), intermittent and irreproducible warnings are also quite dubious. This commit therefore uses a single READ_ONCE() to pick up the value of rhp->func once, then tests that value twice, thus guaranteeing consistent processing within rcu_head_after_call_rcu()().

Neverthless, racing rcu_head_after_call_rcu() with call_rcu() is still a dubious use case.

[ paulmck: Add blank line after declaration per checkpatch.pl. ]
---
 include/linux/rcupdate.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
include/linux/rcupdate.h +4 -2
index 6cdb1db776cf..922bb6848813 100644
@@ -878,9 +878,11 @@ static inline void rcu_head_init(struct rcu_head *rhp)
 static inline bool
 rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f)
 {
-	if (READ_ONCE(rhp->func) == f)
+	rcu_callback_t func = READ_ONCE(rhp->func);
+
+	if (func == f)
 		return true;
-	WARN_ON_ONCE(READ_ONCE(rhp->func) != (rcu_callback_t)~0L);
+	WARN_ON_ONCE(func != (rcu_callback_t)~0L);
 	return false;
 }
 
Signature · 1 line
2.17.1
#14 lore raw

PATCH 14/18 rcu: Update jiffies_to_sched_qs and adjust_jiffies_till_sched_qs() comments

In reply to Paul E. McKenney

This commit better documents the jiffies_to_sched_qs default-value strategy used by adjust_jiffies_till_sched_qs()

---
 kernel/rcu/tree.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +2 -1
index 466299c3d2da..e117732bcd5d 100644
@@ -406,7 +406,7 @@ static bool rcu_kick_kthreads;
  */
 static ulong jiffies_till_sched_qs = ULONG_MAX;
 module_param(jiffies_till_sched_qs, ulong, 0444);
-static ulong jiffies_to_sched_qs; /* Adjusted version of above if not default */
+static ulong jiffies_to_sched_qs; /* See adjust_jiffies_till_sched_qs(). */
 module_param(jiffies_to_sched_qs, ulong, 0444); /* Display only! */
 
 /*
@@ -424,6 +424,7 @@ static void adjust_jiffies_till_sched_qs(void)
 		WRITE_ONCE(jiffies_to_sched_qs, jiffies_till_sched_qs);
 		return;
 	}
+	/* Otherwise, set to third fqs scan, but bound below on large system. */
 	j = READ_ONCE(jiffies_till_first_fqs) +
 		      2 * READ_ONCE(jiffies_till_next_fqs);
 	if (j < HZ / 10 + nr_cpu_ids / RCU_JIFFIES_FQS_DIV)
Signature · 1 line
2.17.1
#15 lore raw

PATCH 15/18 rcu: Fix force_qs_rnp() header comment

In reply to Paul E. McKenney

From: Zhouyi Zhou <[email protected]>

Previously, threads blocked on offlining CPUS were migrated to the root rcu_node structure, thus requiring RCU priority boosting on this structure. However, since commit d19fb8d1f3f6 ("rcu: Don't migrate blocked tasks even if all corresponding CPUs offline"), RCU does not migrate blocked tasks. Consequently, RCU no longer does RCU priority boosting on the root rcu_node structure as of commit 1be0085b515e ("rcu:
Don't initiate RCU priority boosting on root rcu_node").

This commit therefore brings comments for the force_qs_rnp() function's header comment in line with this new no-root-boosting reality.

[ paulmck: Also remove obsolete comment on suppressing new grace periods. ]
---
 kernel/rcu/tree.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/rcu/tree.c +5 -5
index e117732bcd5d..abc8512ceb5f 100644
@@ -2548,11 +2548,11 @@ void rcu_sched_clock_irq(int user)
 }
 
 /*
- * Scan the leaf rcu_node structures, processing dyntick state for any that
- * have not yet encountered a quiescent state, using the function specified.
- * Also initiate boosting for any threads blocked on the root rcu_node.
- *
- * The caller must have suppressed start of new grace periods.
+ * Scan the leaf rcu_node structures.  For each structure on which all
+ * CPUs have reported a quiescent state and on which there are tasks
+ * blocking the current grace period, initiate RCU priority boosting.
+ * Otherwise, invoke the specified function to check dyntick state for
+ * each CPU that has not yet reported a quiescent state.
  */
 static void force_qs_rnp(int (*f)(struct rcu_data *rdp))
 {
Signature · 1 line
2.17.1
#16 lore raw

PATCH 16/18 rcu: Eliminate redundant NULL-pointer check

In reply to Paul E. McKenney

Because rcu_wake_cond() checks for a null task_struct pointer, there is no need for its callers to do so. This commit eliminates the redundant check.

---
 kernel/rcu/tree_plugin.h | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
kernel/rcu/tree_plugin.h +2 -5
index f0aeb7416dcc..81d3cd821891 100644
@@ -1191,8 +1191,6 @@ static int rcu_boost_kthread(void *arg)
 static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
 	__releases(rnp->lock)
 {
-	struct task_struct *t;
-
 	raw_lockdep_assert_held_rcu_node(rnp);
 	if (!rcu_preempt_blocked_readers_cgp(rnp) && rnp->exp_tasks == NULL) {
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
@@ -1206,9 +1204,8 @@ static void rcu_initiate_boost(struct rcu_node *rnp, unsigned long flags)
 		if (rnp->exp_tasks == NULL)
 			rnp->boost_tasks = rnp->gp_tasks;
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
-		t = rnp->boost_kthread_task;
-		if (t)
-			rcu_wake_cond(t, rnp->boost_kthread_status);
+		rcu_wake_cond(rnp->boost_kthread_task,
+			      rnp->boost_kthread_status);
 	} else {
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 	}
Signature · 1 line
2.17.1
#17 lore raw

PATCH 17/18 rcu: Fix typo in tree_exp.h comment

In reply to Paul E. McKenney

This commit changes a rcu_exp_handler() comment from rcu_preempt_defer_qs() to rcu_preempt_deferred_qs() in order to better match reality.

---
 kernel/rcu/tree_exp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_exp.h +1 -1
index 4c2a0189e748..ec4fb93a5dbe 100644
@@ -648,7 +648,7 @@ static void rcu_exp_handler(void *unused)
 	 *
 	 * If the CPU is fully enabled (or if some buggy RCU-preempt
 	 * read-side critical section is being used from idle), just
-	 * invoke rcu_preempt_defer_qs() to immediately report the
+	 * invoke rcu_preempt_deferred_qs() to immediately report the
 	 * quiescent state.  We cannot use rcu_read_unlock_special()
 	 * because we are in an interrupt handler, which will cause that
 	 * function to take an early exit without doing anything.
Signature · 1 line
2.17.1
#18 lore raw

PATCH 18/18 rcu: Correct READ_ONCE()/WRITE_ONCE() for ->rcu_read_unlock_special

In reply to Paul E. McKenney

The task_struct structure's ->rcu_read_unlock_special field is only ever read or written by the owning task, but it is accessed both at process and interrupt levels. It may therefore be accessed using plain reads and writes while interrupts are disabled, but must be accessed using READ_ONCE() and WRITE_ONCE() or better otherwise. This commit makes a few adjustments to align with this discipline.

---
 kernel/rcu/tree_exp.h    | 2 +-
 kernel/rcu/tree_plugin.h | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
kernel/rcu/tree_exp.h +1 -1
index ec4fb93a5dbe..1ee0782213b8 100644
@@ -633,7 +633,7 @@ static void rcu_exp_handler(void *unused)
 		raw_spin_lock_irqsave_rcu_node(rnp, flags);
 		if (rnp->expmask & rdp->grpmask) {
 			rdp->deferred_qs = true;
-			WRITE_ONCE(t->rcu_read_unlock_special.b.exp_hint, true);
+			t->rcu_read_unlock_special.b.exp_hint = true;
 		}
 		raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
 		return;
kernel/rcu/tree_plugin.h +2 -2
index 81d3cd821891..6ddb3c05e88f 100644
@@ -285,7 +285,7 @@ static void rcu_qs(void)
 				       TPS("cpuqs"));
 		__this_cpu_write(rcu_data.cpu_no_qs.b.norm, false);
 		barrier(); /* Coordinate with rcu_flavor_sched_clock_irq(). */
-		current->rcu_read_unlock_special.b.need_qs = false;
+		WRITE_ONCE(current->rcu_read_unlock_special.b.need_qs, false);
 	}
 }
 
@@ -817,7 +817,7 @@ void exit_rcu(void)
 	if (unlikely(!list_empty(&current->rcu_node_entry))) {
 		t->rcu_read_lock_nesting = 1;
 		barrier();
-		t->rcu_read_unlock_special.b.blocked = true;
+		WRITE_ONCE(t->rcu_read_unlock_special.b.blocked, true);
 	} else if (unlikely(t->rcu_read_lock_nesting)) {
 		t->rcu_read_lock_nesting = 1;
 	} else {
Signature · 1 line
2.17.1