Grace-period-related changes for 3.13
cover Grace-period-related changes for 3.13
Hello!
This series contains updates to RCU's grace-period processsing, mostly to improve debuggability of RCU itself.
1. Add consistency checks to the grace-period initialization logic.
2. Prevent spurious-wakeup DoS attack on rcu_gp_kthread.
3. Flag lockless accesses to ->gp_flags with ACCESS_ONCE().
4-6. Add event tracing to various aspects of RCU grace period processing.
7. Distinguish between NOCB and non-NOCB rcu_callback trace events.
8. Add event tracing to track rcu_nocb_kthread() sleeping and awakening.
9. Avoid sparse warnings in rcu_nocb_wake trace event.
10. Make RCU use resched_cpu() instead of the current set_need_resched(),
courtesy of Peter Zijlstra.
Thanx, Paul
b/include/trace/events/rcu.h | 80 +++++++++++++++++++++++++++++++++++-------- b/kernel/rcutiny.c | 2 - b/kernel/rcutree.c | 71 ++++++++++++++++++++++++++++++++------ b/kernel/rcutree_plugin.h | 35 ++++++++++++++++-- b/kernel/sched/core.c | 10 +---- 5 files changed, 160 insertions(+), 38 deletions(-)
PATCH 1/10 rcu: Improve grace-period start logic
From: "Paul E. McKenney" <[email protected]>
This commit improves grace-period start logic by checking ->gp_flags under the lock and by issuing a warning if a grace period is already in progress.
--- kernel/rcutree.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
kernel/rcutree.c +12 -5
@@ -1297,7 +1297,7 @@ static void note_gp_changes(struct rcu_state *rsp, struct rcu_data *rdp) }/* -* Initialize a new grace period. +* Initialize a new grace period. Return 0 if no grace period required. */ static int rcu_gp_init (struct rcu_state *rsp) {
@@ -1306,10 +1306,18 @@ static int rcu_gp_init(struct rcu_state *rsp)rcu_bind_gp_kthread ();raw_spin_lock_irq (&rnp->lock); +if (rsp->gp_flags ==0 ) { +/* Spurious wakeup, tell caller to go back to sleep. */ +raw_spin_unlock_irq (&rnp->lock); +return 0 ; + } rsp->gp_flags =0 ;/* Clear all flags: New grace period. */ -if (rcu_gp_in_progress (rsp)) { -/* Grace period already in progress, don't start another. */ +if (WARN_ON_ONCE (rcu_gp_in_progress (rsp))) { +/* +* Grace period already in progress, don't start another. +* Not supposed to be able to happen. +*/ raw_spin_unlock_irq (&rnp->lock);return 0 ; }
@@ -1474,8 +1482,7 @@ static int __noreturn rcu_gp_kthread(void *arg)wait_event_interruptible (rsp->gp_wq, rsp->gp_flags &RCU_GP_FLAG_INIT ); -if ((rsp->gp_flags &RCU_GP_FLAG_INIT ) && -rcu_gp_init (rsp)) +if (rcu_gp_init (rsp))break ;cond_resched ();flush_signals (current);
Signature · 1 line
1.8.1.5
PATCH 2/10 rcu: Prevent spurious-wakeup DoS attack on rcu_gp_kthread()
From: "Paul E. McKenney" <[email protected]>
Spurious wakeups in the force-quiescent-state loop in rcu_gp_kthread() cause the timeout to be recalculated, which would prevent rcu_gp_fqs() from ever being called. This would in turn would prevent the grace period from ever ending for as long as there was at least one CPU in an extended quiescent state that had not yet passed through a quiescent state.
This commit therefore avoids recalculating the timeout unless the previous pass's call to wait_event_interruptible_timeout() actually did time out, thus preventing the above scenario.
--- kernel/rcutree.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
kernel/rcutree.c +8 -3
@@ -1470,6 +1470,7 @@ static void rcu_gp_cleanup(struct rcu_state *rsp)static int __noreturn rcu_gp_kthread (void *arg) {int fqs_state; +int gf;unsigned long j;int ret;struct rcu_state *rsp = arg;
@@ -1495,10 +1496,13 @@ static int __noreturn rcu_gp_kthread(void *arg) j =HZ ; jiffies_till_first_fqs =HZ ; } + ret =0 ;for (;;) { - rsp->jiffies_force_qs = jiffies + j; +if (!ret) + rsp->jiffies_force_qs = jiffies + j; ret =wait_event_interruptible_timeout (rsp->gp_wq, - (rsp->gp_flags &RCU_GP_FLAG_FQS ) || + ((gf =ACCESS_ONCE (rsp->gp_flags)) & +RCU_GP_FLAG_FQS ) || (!ACCESS_ONCE (rnp->qsmask) && !rcu_preempt_blocked_readers_cgp (rnp)), j);
@@ -1507,7 +1511,8 @@ static int __noreturn rcu_gp_kthread(void *arg) !rcu_preempt_blocked_readers_cgp(rnp))break ;/* If time for quiescent-state forcing, do it. */ -if (ret ==0 || (rsp->gp_flags &RCU_GP_FLAG_FQS )) { +if (ULONG_CMP_GE (jiffies, rsp->jiffies_force_qs) || + (gf &RCU_GP_FLAG_FQS )) { fqs_state =rcu_gp_fqs (rsp, fqs_state);cond_resched (); }else {
Signature · 1 line
1.8.1.5
PATCH 3/10 rcu: Flag lockless access to ->gp_flags with ACCESS_ONCE()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit applies ACCESS_ONCE() to an outside-of-lock access to ->gp_flags. Although it is hard to imagine any sane compiler messing this particular case up, the documentation benefits are substantial. Plus the definition of "sane compiler" grows ever looser.
--- kernel/rcutree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcutree.c +1 -1
@@ -1481,7 +1481,7 @@ static int __noreturn rcu_gp_kthread(void *arg)/* Handle grace-period start. */ for (;;) {wait_event_interruptible (rsp->gp_wq, - rsp->gp_flags & +ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_INIT );if (rcu_gp_init (rsp))break ;
Signature · 1 line
1.8.1.5
PATCH 4/10 rcu: Add tracing to rcu_gp_kthread()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds tracing to the rcu_gp_kthread() function in order to help trace down hangs potentially involving this kthread.
--- include/trace/events/rcu.h | 28 +++++++++++++++++++--------- kernel/rcutree.c | 18 ++++++++++++++++++ 2 files changed, 37 insertions(+), 9 deletions(-)
include/trace/events/rcu.h +19 -9
@@ -39,15 +39,25 @@ TRACE_EVENT(rcu_utilization,#if defined(CONFIG_TREE_RCU ) || defined(CONFIG_TREE_PREEMPT_RCU )/* -* Tracepoint for grace-period events: starting and ending a grace -* period ("start" and "end", respectively), a CPU noting the start -* of a new grace period or the end of an old grace period ("cpustart" -* and "cpuend", respectively), a CPU passing through a quiescent -* state ("cpuqs"), a CPU coming online or going offline ("cpuonl" -* and "cpuofl", respectively), a CPU being kicked for being too -* long in dyntick-idle mode ("kick"), a CPU accelerating its new -* callbacks to RCU_NEXT_READY_TAIL ("AccReadyCB"), and a CPU -* accelerating its new callbacks to RCU_WAIT_TAIL ("AccWaitCB"). +* Tracepoint for grace-period events. Takes a string identifying the +* RCU flavor, the grace-period number, and a string identifying the +* grace-period-related event as follows: +* +* "AccReadyCB": CPU acclerates new callbacks to RCU_NEXT_READY_TAIL. +* "AccWaitCB": CPU accelerates new callbacks to RCU_WAIT_TAIL. +* "start": Start a grace period. +* "cpustart": CPU first notices a grace-period start. +* "cpuqs": CPU passes through a quiescent state. +* "cpuonl": CPU comes online. +* "cpuofl": CPU goes offline. +* "reqwait": GP kthread sleeps waiting for grace-period request. +* "reqwaitsig": GP kthread awakened by signal from reqwait state. +* "fqswait": GP kthread waiting until time to force quiescent states. +* "fqsstart": GP kthread starts forcing quiescent states. +* "fqsend": GP kthread done forcing quiescent states. +* "fqswaitsig": GP kthread awakened by signal from fqswait state. +* "end": End a grace period. +* "cpuend": CPU first notices a grace-period end. */ TRACE_EVENT(rcu_grace_period,
kernel/rcutree.c +18 -0
@@ -1480,6 +1480,9 @@ static int __noreturn rcu_gp_kthread(void *arg)/* Handle grace-period start. */ for (;;) { +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("reqwait" ));wait_event_interruptible (rsp->gp_wq,ACCESS_ONCE (rsp->gp_flags) &RCU_GP_FLAG_INIT );
@@ -1487,6 +1490,9 @@ static int __noreturn rcu_gp_kthread(void *arg)break ;cond_resched ();flush_signals (current); +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("reqwaitsig" )); }/* Handle quiescent-state forcing. */
@@ -1500,6 +1506,9 @@ static int __noreturn rcu_gp_kthread(void *arg)for (;;) {if (!ret) rsp->jiffies_force_qs = jiffies + j; +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("fqswait" )); ret = wait_event_interruptible_timeout(rsp->gp_wq, ((gf = ACCESS_ONCE(rsp->gp_flags)) & RCU_GP_FLAG_FQS) ||
@@ -1513,12 +1522,21 @@ static int __noreturn rcu_gp_kthread(void *arg)/* If time for quiescent-state forcing, do it. */ if (ULONG_CMP_GE (jiffies, rsp->jiffies_force_qs) || (gf &RCU_GP_FLAG_FQS )) { +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("fqsstart" )); fqs_state =rcu_gp_fqs (rsp, fqs_state); +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("fqsend" ));cond_resched (); }else {/* Deal with stray signal. */ cond_resched ();flush_signals (current); +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("fqswaitsig" )); } j = jiffies_till_next_fqs;if (j >HZ ) {
Signature · 1 line
1.8.1.5
PATCH 5/10 rcu: Add tracing of normal (non-NOCB) grace-period requests
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds tracing to the normal grace-period request points. These are rcu_gp_cleanup(), which checks for the need for another grace period at the end of the previous grace period, and rcu_start_gp_advanced(), which restarts RCU's state machine after an idle period. These trace events are intended to help track down bugs where RCU remains idle despite there being work for it to do.
--- include/trace/events/rcu.h | 1 + kernel/rcutree.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +1 -0
@@ -45,6 +45,7 @@ TRACE_EVENT(rcu_utilization,* * "AccReadyCB": CPU acclerates new callbacks to RCU_NEXT_READY_TAIL. * "AccWaitCB": CPU accelerates new callbacks to RCU_WAIT_TAIL. +* "newreq": Request a new grace period. * "start": Start a grace period. * "cpustart": CPU first notices a grace-period start. * "cpuqs": CPU passes through a quiescent state.
kernel/rcutree.c +7 -1
@@ -1459,8 +1459,12 @@ static void rcu_gp_cleanup(struct rcu_state *rsp) rsp->fqs_state =RCU_GP_IDLE ; rdp =this_cpu_ptr (rsp->rda);rcu_advance_cbs (rsp, rnp, rdp);/* Reduce false positives below. */ -if (cpu_needs_another_gp (rsp, rdp)) +if (cpu_needs_another_gp (rsp, rdp)) { rsp->gp_flags =1 ; +trace_rcu_grace_period (rsp->name, +ACCESS_ONCE (rsp->gpnum), +TPS ("newreq" )); + }raw_spin_unlock_irq (&rnp->lock); }
@@ -1584,6 +1588,8 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp,return ; } rsp->gp_flags =RCU_GP_FLAG_INIT ; +trace_rcu_grace_period (rsp->name,ACCESS_ONCE (rsp->gpnum), +TPS ("newreq" ));/* * We can't do wakeups while holding the rnp->lock, as that
Signature · 1 line
1.8.1.5
PATCH 6/10 rcu: Add tracing for rcuo no-CBs CPU wakeup handshake
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Lost wakeups from call_rcu() to the rcuo kthreads can result in hangs that are difficult to diagnose. This commit therefore adds tracing to help pin down the cause of these hangs.
--- include/trace/events/rcu.h | 37 +++++++++++++++++++++++++++++++++++++ kernel/rcutree_plugin.h | 14 +++++++++++++- 2 files changed, 50 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +37 -0
@@ -172,6 +172,42 @@ TRACE_EVENT(rcu_grace_period_init, );/* +* Tracepoint for RCU no-CBs CPU callback handoffs. This event is intended +* to assist debugging of these handoffs. +* +* The first argument is the name of the RCU flavor, and the second is +* the number of the offloaded CPU are extracted. The third and final +* argument is a string as follows: +* +* "WakeEmpty": Wake rcuo kthread, first CB to empty list. +* "WakeOvf": Wake rcuo kthread, CB list is huge. +* "WakeNot": Don't wake rcuo kthread. +* "WakeNotPoll": Don't wake rcuo kthread because it is polling. +* "WokeEmpty": rcuo kthread woke to find empty list. +* "WokeNonEmpty": rcuo kthread woke to find non-empty list. +*/ +TRACE_EVENT (rcu_nocb_wake, + + TP_PROTO(const char *rcuname, int cpu, const char *reason), + +TP_ARGS (rcuname, cpu, reason), + +TP_STRUCT__entry ( + __field(const char *, rcuname) + __field(int, cpu) + __field(const char *, reason) + ), + +TP_fast_assign ( + __entry->rcuname = rcuname; + __entry->cpu = cpu; + __entry->reason = reason; + ), + +TP_printk ("%s %d %s" , __entry->rcuname, __entry->cpu, __entry->reason) +); + +/* * Tracepoint for tasks blocking within preemptible-RCU read-side * critical sections. Track the type of RCU (which one day might * include SRCU), the grace-period number that the task is blocking
@@ -667,6 +703,7 @@ TRACE_EVENT(rcu_barrier,#define trace_rcu_future_grace_period (rcuname, gpnum, completed, c, \ level, grplo, grphi, event) \ do { } while (0) +#define trace_rcu_nocb_wake (rcuname, cpu, reason) do { } while (0)#define trace_rcu_preempt_task (rcuname, pid, gpnum) do { } while (0)#define trace_rcu_unlock_preempted_task (rcuname, gpnum, pid) do { } while (0) #define trace_rcu_quiescent_state_report(rcuname, gpnum, mask, qsmask, level, \
kernel/rcutree_plugin.h +13 -1
@@ -2108,15 +2108,22 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,/* If we are not being polled and there is a kthread, awaken it ... */ t =ACCESS_ONCE (rdp->nocb_kthread); -if (rcu_nocb_poll | !t) +if (rcu_nocb_poll | !t) { +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("WakeNotPoll" ));return ; + } len =atomic_long_read (&rdp->nocb_q_count);if (old_rhpp == &rdp->nocb_head) {wake_up (&rdp->nocb_wq);/* ... only if queue was empty ... */ rdp->qlen_last_fqs_check =0 ; +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("WakeEmpty" )); }else if (len > rdp->qlen_last_fqs_check + qhimark) {wake_up_process (t);/* ... or if many callbacks queued. */ rdp->qlen_last_fqs_check =LONG_MAX /2 ; +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("WakeOvf" )); + }else { +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("WakeNot" )); }return ; }
@@ -2233,10 +2240,15 @@ static int rcu_nocb_kthread(void *arg)wait_event_interruptible (rdp->nocb_wq, rdp->nocb_head); list =ACCESS_ONCE (rdp->nocb_head);if (!list) { +if (!rcu_nocb_poll) +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("WokeEmpty" ));schedule_timeout_interruptible (1 );flush_signals (current);continue ; } +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("WokeNonEmpty" ));/* * Extract queued callbacks, update counts, and wait
Signature · 1 line
1.8.1.5
PATCH 7/10 rcu: Distinguish between NOCB and non-NOCB rcu_callback trace events
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
One way to distinguish between NOCB and non-NOCB rcu_callback trace events is that the former always print zero for the lazy and non-lazy queue lengths. Unfortunately, this also means that we cannot see the NOCB queue lengths. This commit therefore accesses the NOCB queue lengths, but negates them. NOCB rcu_callback trace events should therefore have negative queue lengths.
--- kernel/rcutree_plugin.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/rcutree_plugin.h +4 -2
@@ -2147,10 +2147,12 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,if (__is_kfree_rcu_offset ((unsigned long )rhp->func))trace_rcu_kfree_callback (rdp->rsp->name, rhp, (unsigned long )rhp->func, - rdp->qlen_lazy, rdp->qlen); + -atomic_long_read (&rdp->nocb_q_count_lazy), + -atomic_long_read (&rdp->nocb_q_count));else trace_rcu_callback (rdp->rsp->name, rhp, - rdp->qlen_lazy, rdp->qlen); + -atomic_long_read (&rdp->nocb_q_count_lazy), + -atomic_long_read (&rdp->nocb_q_count));return 1 ; }
Signature · 1 line
1.8.1.5
PATCH 8/10 rcu: Track rcu_nocb_kthread()'s sleeping and awakening
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds event traces to track all of rcu_nocb_kthread()'s blocking and awakening.
--- include/trace/events/rcu.h | 4 ++++ kernel/rcutree_plugin.h | 15 ++++++++++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
include/trace/events/rcu.h +4 -0
@@ -183,8 +183,12 @@ TRACE_EVENT(rcu_grace_period_init,* "WakeOvf": Wake rcuo kthread, CB list is huge. * "WakeNot": Don't wake rcuo kthread. * "WakeNotPoll": Don't wake rcuo kthread because it is polling. +* "Poll": Start of new polling cycle for rcu_nocb_poll. +* "Sleep": Sleep waiting for CBs for !rcu_nocb_poll. * "WokeEmpty": rcuo kthread woke to find empty list. * "WokeNonEmpty": rcuo kthread woke to find non-empty list. +* "WaitQueue": Enqueue partially done, timed wait for it to complete. +* "WokeQueue": Partial enqueue now complete. */ TRACE_EVENT(rcu_nocb_wake,
kernel/rcutree_plugin.h +14 -1
@@ -2230,6 +2230,7 @@ static void rcu_nocb_wait_gp(struct rcu_data *rdp)static int rcu_nocb_kthread (void *arg) {int c, cl; +bool firsttime =1 ;struct rcu_head *list;struct rcu_head *next;struct rcu_head **tail;
@@ -2238,8 +2239,15 @@ static int rcu_nocb_kthread(void *arg)/* Each pass through this loop invokes one batch of callbacks */ for (;;) {/* If not polling, wait for next batch of callbacks. */ -if (!rcu_nocb_poll) +if (!rcu_nocb_poll) { +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("Sleep" ));wait_event_interruptible (rdp->nocb_wq, rdp->nocb_head); + }else if (firsttime) { + firsttime =0 ; +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("Poll" )); + } list =ACCESS_ONCE (rdp->nocb_head);if (!list) {if (!rcu_nocb_poll)
@@ -2249,6 +2257,7 @@ static int rcu_nocb_kthread(void *arg)flush_signals (current);continue ; } + firsttime =1 ;trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu,TPS ("WokeNonEmpty" ));
@@ -2271,7 +2280,11 @@ static int rcu_nocb_kthread(void *arg) next = list->next;/* Wait for enqueuing to complete, if needed. */ while (next ==NULL && &list->next != tail) { +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("WaitQueue" ));schedule_timeout_interruptible (1 ); +trace_rcu_nocb_wake (rdp->rsp->name, rdp->cpu, +TPS ("WokeQueue" )); next = list->next; }debug_rcu_head_unqueue (list);
Signature · 1 line
1.8.1.5
PATCH 9/10 rcu: Avoid sparse warnings in rcu_nocb_wake trace event
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The event-tracing macros do not like bool tracing arguments, so this commit makes them be of type char. This change has the knock-on effect of making it illegal to pass a pointer into one of these arguments, so also change rcutiny's first call to trace_rcu_batch_end() to convert from pointer to boolean, prefixing with "!!".
--- include/trace/events/rcu.h | 10 +++++----- kernel/rcutiny.c | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-)
include/trace/events/rcu.h +5 -5
@@ -591,17 +591,17 @@ TRACE_EVENT(rcu_invoke_kfree_callback, TRACE_EVENT(rcu_batch_end, TP_PROTO(const char *rcuname, int callbacks_invoked, - bool cb, bool nr, bool iit, bool risk), + char cb, char nr, char iit, char risk), TP_ARGS(rcuname, callbacks_invoked, cb, nr, iit, risk), TP_STRUCT__entry( __field(const char *, rcuname) __field(int, callbacks_invoked) - __field(bool, cb) - __field(bool, nr) - __field(bool, iit) - __field(bool, risk) + __field(char, cb) + __field(char, nr) + __field(char, iit) + __field(char, risk) ), TP_fast_assign(
kernel/rcutiny.c +1 -1
@@ -273,7 +273,7 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp)if (&rcp->rcucblist == rcp->donetail) {RCU_TRACE (trace_rcu_batch_start (rcp->name,0 ,0 ,-1 ));RCU_TRACE (trace_rcu_batch_end (rcp->name,0 , -ACCESS_ONCE (rcp->rcucblist), + !!ACCESS_ONCE (rcp->rcucblist),need_resched (),is_idle_task (current), false));
Signature · 1 line
1.8.1.5
PATCH 10/10 sched, rcu: Make RCU use resched_cpu()
In reply to Paul E. McKenneyFrom: Peter Zijlstra <[email protected]>
We're going to deprecate and remove set_need_resched() for it will do the wrong thing. Make an exception for RCU and allow it to use resched_cpu() which will do the right thing.
--- kernel/rcutree.c | 15 ++++++++++++++- kernel/sched/core.c | 10 ++-------- 2 files changed, 16 insertions(+), 9 deletions(-)
kernel/rcutree.c +14 -1
@@ -898,6 +898,12 @@ static void print_other_cpu_stall(struct rcu_state *rsp)force_quiescent_state (rsp);/* Kick them all. */ } +/* +* This function really isn't for public consumption, but RCU is special in +* that context switches can allow the state machine to make progress. +*/ +extern void resched_cpu (int cpu); +static void print_cpu_stall (struct rcu_state *rsp) {int cpu;
@@ -927,7 +933,14 @@ static void print_cpu_stall(struct rcu_state *rsp)3 *rcu_jiffies_till_stall_check () +3 ;raw_spin_unlock_irqrestore (&rnp->lock, flags); -set_need_resched ();/* kick ourselves to get things going. */ +/* +* Attempt to revive the RCU machinery by forcing a context switch. +* +* A context switch would normally allow the RCU state machine to make +* progress and it could be we're stuck in kernel space without context +* switches for an entirely unreasonable amount of time. +*/ +resched_cpu (smp_processor_id ()); }static void check_cpu_stall (struct rcu_state *rsp,struct rcu_data *rdp)
kernel/sched/core.c +2 -8
@@ -513,12 +513,11 @@ static inline void init_hrtick(void)* might also involve a cross-CPU call to trigger the scheduler on * the target CPU. */ -#ifdef CONFIG_SMP void resched_task (struct task_struct *p) {int cpu; -assert_raw_spin_locked (&task_rq (p)->lock); +lockdep_assert_held (&task_rq (p)->lock);if (test_tsk_need_resched (p))return ;
@@ -546,6 +545,7 @@ void resched_cpu(int cpu)raw_spin_unlock_irqrestore (&rq->lock, flags); } +#ifdef CONFIG_SMP #ifdef CONFIG_NO_HZ_COMMON /* * In the semi idle case, use the nearest busy cpu for migrating timers
@@ -693,12 +693,6 @@ void sched_avg_update(struct rq *rq) } } -#else /* !CONFIG_SMP */ -void resched_task (struct task_struct *p) -{ -assert_raw_spin_locked (&task_rq (p)->lock); -set_tsk_need_resched (p); -}#endif /* CONFIG_SMP */ #if defined(CONFIG_RT_GROUP_SCHED) || (defined(CONFIG_FAIR_GROUP_SCHED) && \
Signature · 1 line
1.8.1.5