Remove scheduler restriction, update for v5.8
Subsystems (3)
Paul E. McKenney
maintainercover Remove scheduler restriction, update for v5.8
Hello!
This series removes the long-standing restriction preventing holding of scheduler locks across rcu_read_unlock() unless interrupts have been disabled across the entire RCU read-side critical section.
For example, with this series, the following code is now perfectly legal, and is in fact tested by rcutorture:
// Preemption is enabled. rcu_read_lock(); do_something(); raw_spin_lock_irqsave(¤t->pi_lock, flags); rcu_read_unlock(); raw_spin_unlock_irqrestore(¤t->pi_lock, flags);
This change was enabled by the earlier changes that turn on scheduler-clock interrupts for nohz_full CPUs that take too long to report a quiescent state during a given grace period. (Yes, nohz_full CPUs executing in userspace still have their quiescent states implicitly reported by the RCU grace-period kthread!)
And here is the series:
1. Add KCSAN stubs to update.c.
2. Make rcu_read_unlock_special() safe for rq/pi locks.
3. Don't set nesting depth negative in rcu_preempt_deferred_qs(),
courtesy of Lai Jiangshan.
4. Remove unused ->rcu_read_unlock_special.b.deferred_qs field,
courtesy of Lai Jiangshan.
5. Don't use negative nesting depth in __rcu_read_unlock(),
courtesy of Lai Jiangshan.
6. Add test of holding scheduler locks across rcu_read_unlock().
Thanx, Paul
------------------------------------------------------------------------
include/linux/sched.h | 2 +- kernel/rcu/rcutorture.c | 10 +++++++++- kernel/rcu/tree_exp.h | 31 +++++-------------------------- kernel/rcu/tree_plugin.h | 45 +++++++++++++++------------------------------ kernel/rcu/update.c | 13 +++++++++++++ 5 files changed, 43 insertions(+), 58 deletions(-)
[email protected]
maintainerPATCH 1/6 rcu: Add KCSAN stubs to update.c
From: "Paul E. McKenney" <[email protected]>
This commit adds stubs for KCSAN's data_race(), ASSERT_EXCLUSIVE_WRITER(), and ASSERT_EXCLUSIVE_ACCESS() macros to allow code using these macros to move ahead.
--- kernel/rcu/update.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
kernel/rcu/update.c +13 -0
@@ -51,6 +51,19 @@#endif #define MODULE_PARAM_PREFIX "rcupdate." +#ifndef data_race +#define data_race (expr) \ + ({ \ + expr; \ + }) +#endif +#ifndef ASSERT_EXCLUSIVE_WRITER +#define ASSERT_EXCLUSIVE_WRITER (var) do { } while (0) +#endif +#ifndef ASSERT_EXCLUSIVE_ACCESS +#define ASSERT_EXCLUSIVE_ACCESS (var) do { } while (0) +#endif +#ifndef CONFIG_TINY_RCU module_param (rcu_expedited, int,0 );module_param (rcu_normal, int,0 );
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 2/6 rcu: Make rcu_read_unlock_special() safe for rq/pi locks
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The scheduler is currently required to hold rq/pi locks across the entire RCU read-side critical section or not at all. This is inconvenient and leaves traps for the unwary, including the author of this commit.
But now that excessively ong grace periods enable scheduling-clock interrupts for holdout nohz_full CPUs, the nohz_full rescue logic in rcu_read_unlock_special() can be dispensed with. In other words, the rcu_read_unlock_special() function can refrain from doing wakeups unless such wakeups are guaranteed safe.
This commit therefore avoids unsafe wakeups, freeing the scheduler to hold rq/pi locks across rcu_read_unlock() even if the corresponding RCU read-side critical section might have been preempted.
This commit is inspired by a patch from Lai Jiangshan:
https://lore.kernel.org/lkml/[email protected]
This commit is further intended to be a step towards his goal of permitting the inlining of RCU-preempt's rcu_read_lock() and rcu_read_unlock().
--- kernel/rcu/tree_plugin.h | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-)
kernel/rcu/tree_plugin.h +8 -9
@@ -615,19 +615,18 @@ static void rcu_read_unlock_special(struct task_struct *t)struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; - exp = (t->rcu_blocked_node && t->rcu_blocked_node->exp_tasks) || - (rdp->grpmask &READ_ONCE (rnp->expmask)) || -tick_nohz_full_cpu (rdp->cpu); + exp = (t->rcu_blocked_node && +READ_ONCE (t->rcu_blocked_node->exp_tasks)) || + (rdp->grpmask &READ_ONCE (rnp->expmask));// Need to defer quiescent state until everything is enabled. -if (irqs_were_disabled && use_softirq && - (in_interrupt () || - (exp && !t->rcu_read_unlock_special.b.deferred_qs))) { -// Using softirq, safe to awaken, and we get -// no help from enabling irqs, unlike bh/preempt. +if (use_softirq && (in_irq () || (exp && !irqs_were_disabled))) { +// Using softirq, safe to awaken, and either the +// wakeup is free or there is an expedited GP. raise_softirq_irqoff (RCU_SOFTIRQ ); }else {// Enabling BH or preempt does reschedule, so... -// Also if no expediting or NO_HZ_FULL, slow is OK. +// Also if no expediting, slow is OK. +// Plus nohz_full CPUs eventually get tick enabled. set_tsk_need_resched (current);set_preempt_need_resched (); if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 3/6 rcu: Don't set nesting depth negative in rcu_preempt_deferred_qs()
In reply to Paul E. McKenneyFrom: Lai Jiangshan <[email protected]>
Now that RCU flavors have been consolidated, an RCU-preempt rcu_read_unlock() in an interrupt or softirq handler cannot possibly end the RCU read-side critical section. Consider the old vulnerability involving rcu_preempt_deferred_qs() being invoked within such a handler that interrupted an extended RCU read-side critical section, in which a wakeup might be invoked with a scheduler lock held. Because rcu_read_unlock_special() no longer does wakeups in such situations, it is no longer necessary for rcu_preempt_deferred_qs() to set the nesting level negative.
This commit therfore removes this recursion-protection code from rcu_preempt_deferred_qs().
--- kernel/rcu/tree_plugin.h | 5 ----- 1 file changed, 5 deletions(-)
kernel/rcu/tree_plugin.h +0 -5
@@ -569,16 +569,11 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t)static void rcu_preempt_deferred_qs (struct task_struct *t) {unsigned long flags; -bool couldrecurse =rcu_preempt_depth () >=0 ;if (!rcu_preempt_need_deferred_qs (t))return ; -if (couldrecurse) -rcu_preempt_depth_set (rcu_preempt_depth () -RCU_NEST_BIAS );local_irq_save (flags);rcu_preempt_deferred_qs_irqrestore (t, flags); -if (couldrecurse) -rcu_preempt_depth_set (rcu_preempt_depth () +RCU_NEST_BIAS ); }/*
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 4/6 rcu: Remove unused ->rcu_read_unlock_special.b.deferred_qs field
In reply to Paul E. McKenneyFrom: Lai Jiangshan <[email protected]>
The ->rcu_read_unlock_special.b.deferred_qs field is set to true in rcu_read_unlock_special() but never set to false. This is not particularly useful, so this commit removes this field.
The only possible justification for this field is to ease debugging of RCU deferred quiscent states, but the combination of the other ->rcu_read_unlock_special fields plus ->rcu_blocked_node and of course ->rcu_read_lock_nesting should cover debugging needs. And if this last proves incorrect, this patch can always be reverted, along with the required setting of ->rcu_read_unlock_special.b.deferred_qs to false in rcu_preempt_deferred_qs_irqrestore().
--- include/linux/sched.h | 2 +- kernel/rcu/tree_plugin.h | 1 - 2 files changed, 1 insertion(+), 2 deletions(-)
include/linux/sched.h +1 -1
@@ -613,7 +613,7 @@ union rcu_special {u8 blocked;u8 need_qs;u8 exp_hint;/* Hint for performance. */ -u8 deferred_qs; +u8 pad;/* No garbage from compiler! */ } b;/* Bits. */ u32 s;/* Set of bits. */ };
kernel/rcu/tree_plugin.h +0 -1
@@ -634,7 +634,6 @@ static void rcu_read_unlock_special(struct task_struct *t)irq_work_queue_on (&rdp->defer_qs_iw, rdp->cpu); } } - t->rcu_read_unlock_special.b.deferred_qs = true;local_irq_restore (flags);return ; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 5/6 rcu: Don't use negative nesting depth in __rcu_read_unlock()
In reply to Paul E. McKenneyFrom: Lai Jiangshan <[email protected]>
Now that RCU flavors have been consolidated, an RCU-preempt rcu_read_unlock() in an interrupt or softirq handler cannot possibly end the RCU read-side critical section. Consider the old vulnerability involving rcu_read_unlock() being invoked within such a handler that interrupted an __rcu_read_unlock_special(), in which a wakeup might be invoked with a scheduler lock held. Because rcu_read_unlock_special() no longer does wakeups in such situations, it is no longer necessary for __rcu_read_unlock() to set the nesting level negative.
This commit therfore removes this recursion-protection code from __rcu_read_unlock().
--- kernel/rcu/tree_exp.h | 31 +++++-------------------------- kernel/rcu/tree_plugin.h | 22 +++++++--------------- 2 files changed, 12 insertions(+), 41 deletions(-)
kernel/rcu/tree_exp.h +5 -26
@@ -639,6 +639,7 @@ static void wait_rcu_exp_gp(struct work_struct *wp)*/ static void rcu_exp_handler (void *unused) { +int depth =rcu_preempt_depth ();unsigned long flags;struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode;
@@ -649,7 +650,7 @@ static void rcu_exp_handler(void *unused)* critical section. If also enabled or idle, immediately * report the quiescent state, otherwise defer. */ -if (!rcu_preempt_depth ()) { +if (!depth) {if (!(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK )) ||rcu_dynticks_curr_cpu_in_eqs ()) {rcu_report_exp_rdp (rdp);
@@ -673,7 +674,7 @@ static void rcu_exp_handler(void *unused)* can have caused this quiescent state to already have been * reported, so we really do need to check ->expmask. */ -if (rcu_preempt_depth () >0 ) { +if (depth >0 ) {raw_spin_lock_irqsave_rcu_node (rnp, flags);if (rnp->expmask & rdp->grpmask) { rdp->exp_deferred_qs = true;
@@ -683,30 +684,8 @@ static void rcu_exp_handler(void *unused)return ; } -/* -* The final and least likely case is where the interrupted -* code was just about to or just finished exiting the RCU-preempt -* read-side critical section, and no, we can't tell which. -* So either way, set ->deferred_qs to flag later code that -* a quiescent state is required. -* -* If the CPU is fully enabled (or if some buggy RCU-preempt -* read-side critical section is being used from idle), just -* invoke rcu_preempt_deferred_qs() to immediately report the -* quiescent state. We cannot use rcu_read_unlock_special() -* because we are in an interrupt handler, which will cause that -* function to take an early exit without doing anything. -* -* Otherwise, force a context switch after the CPU enables everything. -*/ - rdp->exp_deferred_qs = true; -if (!(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK )) || -WARN_ON_ONCE (rcu_dynticks_curr_cpu_in_eqs ())) { -rcu_preempt_deferred_qs (t); - }else { -set_tsk_need_resched (t); -set_preempt_need_resched (); - } +// Finally, negative nesting depth should not happen. +WARN_ON_ONCE (1 ); }/* PREEMPTION=y, so no PREEMPTION=n expedited grace period to clean up after. */
kernel/rcu/tree_plugin.h +7 -15
@@ -345,9 +345,7 @@ static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp)return READ_ONCE (rnp->gp_tasks) !=NULL ; } -/* Bias and limit values for ->rcu_read_lock_nesting. */ -#define RCU_NEST_BIAS INT_MAX -#define RCU_NEST_NMAX (-INT_MAX / 2) +/* limit value for ->rcu_read_lock_nesting. */ #define RCU_NEST_PMAX (INT_MAX / 2)static void rcu_preempt_read_enter (void )
@@ -355,9 +353,9 @@ static void rcu_preempt_read_enter(void) current->rcu_read_lock_nesting++; } -static void rcu_preempt_read_exit (void ) +static int rcu_preempt_read_exit (void ) { - current->rcu_read_lock_nesting--; +return --current->rcu_read_lock_nesting; }static void rcu_preempt_depth_set (int val)
@@ -390,21 +388,15 @@ void __rcu_read_unlock(void) {struct task_struct *t = current; -if (rcu_preempt_depth () !=1 ) { -rcu_preempt_read_exit (); - }else { +if (rcu_preempt_read_exit () ==0 ) {barrier ();/* critical section before exit code. */ -rcu_preempt_depth_set (-RCU_NEST_BIAS ); -barrier ();/* assign before ->rcu_read_unlock_special load */ if (unlikely (READ_ONCE (t->rcu_read_unlock_special.s)))rcu_read_unlock_special (t); -barrier ();/* ->rcu_read_unlock_special load before assign */ -rcu_preempt_depth_set (0 ); }if (IS_ENABLED (CONFIG_PROVE_LOCKING )) {int rrln =rcu_preempt_depth (); -WARN_ON_ONCE (rrln <0 && rrln >RCU_NEST_NMAX ); +WARN_ON_ONCE (rrln <0 || rrln >RCU_NEST_PMAX ); } }EXPORT_SYMBOL_GPL (__rcu_read_unlock);
@@ -556,7 +548,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t) {return (__this_cpu_read (rcu_data.exp_deferred_qs) ||READ_ONCE (t->rcu_read_unlock_special.s)) && -rcu_preempt_depth () <=0 ; +rcu_preempt_depth () ==0 ; }/*
@@ -692,7 +684,7 @@ static void rcu_flavor_sched_clock_irq(int user) }else if (rcu_preempt_need_deferred_qs (t)) {rcu_preempt_deferred_qs (t);/* Report deferred QS. */ return ; - }else if (!rcu_preempt_depth ()) { + }else if (!WARN_ON_ONCE (rcu_preempt_depth ())) {rcu_qs ();/* Report immediate QS. */ return ; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 6/6 rcutorture: Add test of holding scheduler locks across rcu_read_unlock()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Now that it should be safe to hold scheduler locks across rcu_read_unlock(), even in cases where the corresponding RCU read-side critical section might have been preempted and boosted, the commit adds a test of this capability to rcutorture. This has been tested on current mainline (which can deadlock in this situation), and lockdep duly reported the expected deadlock. On -rcu, lockdep is silent, thus far, anyway.
--- kernel/rcu/rcutorture.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
kernel/rcu/rcutorture.c +9 -1
@@ -1147,6 +1147,7 @@ static void rcutorture_one_extend(int *readstate, int newstate,struct torture_random_state *trsp, struct rt_read_seg *rtrsp) { + unsigned long flags;int idxnew =-1 ;int idxold = *readstate;int statesnew = ~*readstate & newstate;
@@ -1181,8 +1182,15 @@ static void rcutorture_one_extend(int *readstate, int newstate,rcu_read_unlock_bh ();if (statesold &RCUTORTURE_RDR_SCHED )rcu_read_unlock_sched (); -if (statesold &RCUTORTURE_RDR_RCU ) +if (statesold &RCUTORTURE_RDR_RCU ) { +bool lockit = !statesnew && !(torture_random (trsp) &0xffff ); + +if (lockit) +raw_spin_lock_irqsave (¤t->pi_lock, flags); cur_ops->readunlock (idxold >>RCUTORTURE_RDR_SHIFT ); +if (lockit) +raw_spin_unlock_irqrestore (¤t->pi_lock, flags); + }/* Delay if neither beginning nor end and there was a change. */ if ((statesnew || statesold) && *readstate && newstate)
Signature · 1 line
2.9.5