rcu: Break call_rcu() deadlock involving scheduler and perf
A message that is not in the archive (sent off-list or not yet received): [email protected]
Peter Zijlstra
maintainerQuoted 17 lines · tip-bot for Paul E. McKenney
The underlying problem is that perf is invoking
call_rcu()with the scheduler locks held, but in NOCB mode,call_rcu()will with high probability invoke the scheduler -- which just might want to use its locks. The reason thatcall_rcu()needs to invoke the scheduler is to wake up the corresponding rcuo callback-offload kthread, which does the job of starting up a grace period and invoking the callbacks afterwards.One solution (championed on a related problem by Lai Jiangshan) is to simply defer the wakeup to some point where scheduler locks are no longer held. Since we don't want to unnecessarily incur the cost of such deferral, the task before us is threefold:
1. Determine when it is likely that a relevant scheduler lock is held.
2. Defer the wakeup in such cases.
Quoted 7 lines
3. Ensure that all deferred wakeups eventually happen, preferably
sooner rather than later.We use
irqs_disabled_flags()as a proxy for relevant scheduler locks being held. This works because the relevant locks are always acquired with interrupts disabled. We may defer more often than needed, but that is at least safe.
This would also allow us to do away with things like the below patch, right?
---
commit 058ebd0eba3aff16b144eabf4510ed9510e1416e
Author: Peter Zijlstra <[email protected]> Date: Fri Jul 12 11:08:33 2013 +0200
perf: Fix perf_lock_task_context() vs RCU
Jiri managed to trigger this warning:
[] ======================================================
[] [ INFO: possible circular locking dependency detected ]
[] 3.10.0+ #228 Tainted: G W
[] -------------------------------------------------------
[] p/6613 is trying to acquire lock:
[] (rcu_node_0){..-...}, at: [<ffffffff810ca797>] rcu_read_unlock_special+0xa7/0x250
[]
[] but task is already holding lock:
[] (&ctx->lock){-.-...}, at: [<ffffffff810f2879>] perf_lock_task_context+0xd9/0x2c0
[]
[] which lock already depends on the new lock.
[]
[] the existing dependency chain (in reverse order) is:
[]
[] -> #4 (&ctx->lock){-.-...}:
[] -> #3 (&rq->lock){-.-.-.}:
[] -> #2 (&p->pi_lock){-.-.-.}:
[] -> #1 (&rnp->nocb_gp_wq[1]){......}:
[] -> #0 (rcu_node_0){..-...}:
Paul was quick to explain that due to preemptible RCU we cannot call
rcu_read_unlock() while holding scheduler (or nested) locks when part
of the read side critical section was preemptible.
Therefore solve it by making the entire RCU read side non-preemptible.
Also pull out the retry from under the non-preempt to play nice with RT.
Reported-by: Jiri Olsa <[email protected]>
Helped-out-by: Paul E. McKenney <[email protected]>
Cc: <[email protected]>
Signed-off-by: Peter Zijlstra <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>kernel/events/core.c +14 -1
@@ -947,8 +947,18 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags) {struct perf_event_context *ctx; -rcu_read_lock (); retry: +/* +* One of the few rules of preemptible RCU is that one cannot do +* rcu_read_unlock() while holding a scheduler (or nested) lock when +* part of the read side critical section was preemptible -- see +* rcu_read_unlock_special(). +* +* Since ctx->lock nests under rq->lock we must ensure the entire read +* side critical section is non-preemptible. +*/ +preempt_disable (); +rcu_read_lock (); ctx =rcu_dereference (task->perf_event_ctxp[ctxn]);if (ctx) {/*
@@ -964,6 +974,8 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags)raw_spin_lock_irqsave (&ctx->lock, *flags);if (ctx !=rcu_dereference (task->perf_event_ctxp[ctxn])) {raw_spin_unlock_irqrestore (&ctx->lock, *flags); +rcu_read_unlock (); +preempt_enable (); goto retry; }
@@ -973,6 +985,7 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags) } }rcu_read_unlock (); +preempt_enable ();return ctx; }
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · tip-bot for Paul E. McKenney
The underlying problem is that perf is invoking
call_rcu()with the scheduler locks held, but in NOCB mode,call_rcu()will with high probability invoke the scheduler -- which just might want to use its locks. The reason thatcall_rcu()needs to invoke the scheduler is to wake up the corresponding rcuo callback-offload kthread, which does the job of starting up a grace period and invoking the callbacks afterwards.One solution (championed on a related problem by Lai Jiangshan) is to simply defer the wakeup to some point where scheduler locks are no longer held. Since we don't want to unnecessarily incur the cost of such deferral, the task before us is threefold:
1. Determine when it is likely that a relevant scheduler lock is held.
2. Defer the wakeup in such cases.
3. Ensure that all deferred wakeups eventually happen, preferably
sooner rather than later.
Quoted 7 lines
Quoted 4 lines
We use
irqs_disabled_flags()as a proxy for relevant scheduler locks being held. This works because the relevant locks are always acquired with interrupts disabled. We may defer more often than needed, but that is at least safe.This would also allow us to do away with things like the below patch, right?
It takes care of one problem, but there are others, including rcu_read_unlock() inovking the scheduler to deboost itself. So for the moment, we still need the below patch.
Thanx, Paul
Quoted 85 lines
---
commit058ebd0eba3aff16b144eabf4510ed9510e1416eAuthor: Peter Zijlstra <[email protected]> Date: Fri Jul 12 11:08:33 2013 +0200perf: Fix perf_lock_task_context() vs RCU Jiri managed to trigger this warning: [] ====================================================== [] [ INFO: possible circular locking dependency detected ] [] 3.10.0+ #228 Tainted: G W [] ------------------------------------------------------- [] p/6613 is trying to acquire lock: [] (rcu_node_0){..-...}, at: [<ffffffff810ca797>] rcu_read_unlock_special+0xa7/0x250 [] [] but task is already holding lock: [] (&ctx->lock){-.-...}, at: [<ffffffff810f2879>] perf_lock_task_context+0xd9/0x2c0 [] [] which lock already depends on the new lock. [] [] the existing dependency chain (in reverse order) is: [] [] -> #4 (&ctx->lock){-.-...}: [] -> #3 (&rq->lock){-.-.-.}: [] -> #2 (&p->pi_lock){-.-.-.}: [] -> #1 (&rnp->nocb_gp_wq[1]){......}: [] -> #0 (rcu_node_0){..-...}: Paul was quick to explain that due to preemptible RCU we cannot call rcu_read_unlock() while holding scheduler (or nested) locks when part of the read side critical section was preemptible. Therefore solve it by making the entire RCU read side non-preemptible. Also pull out the retry from under the non-preempt to play nice with RT. Reported-by: Jiri Olsa <[email protected]> Helped-out-by: Paul E. McKenney <[email protected]> Cc: <[email protected]> Signed-off-by: Peter Zijlstra <[email protected]> Signed-off-by: Ingo Molnar <[email protected]>kernel/events/core.c +14
-1@@ -947,8 +947,18 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags) {struct perf_event_context *ctx; -rcu_read_lock (); retry: +/* +* One of the few rules of preemptible RCU is that one cannot do +* rcu_read_unlock() while holding a scheduler (or nested) lock when +* part of the read side critical section was preemptible -- see +* rcu_read_unlock_special(). +* +* Since ctx->lock nests under rq->lock we must ensure the entire read +* side critical section is non-preemptible. +*/ +preempt_disable (); +rcu_read_lock (); ctx =rcu_dereference (task->perf_event_ctxp[ctxn]);if (ctx) {/* @@ -964,6 +974,8 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags)raw_spin_lock_irqsave (&ctx->lock, *flags);if (ctx !=rcu_dereference (task->perf_event_ctxp[ctxn])) {raw_spin_unlock_irqrestore (&ctx->lock, *flags); +rcu_read_unlock (); +preempt_enable (); goto retry; }@@ -973,6 +985,7 @@ perf_lock_task_context(struct task_struct *task, int ctxn, unsigned long *flags) } }rcu_read_unlock (); +preempt_enable ();return ctx; }
Peter Zijlstra
maintainerQuoted 27 lines · Paul E. McKenney
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · tip-bot for Paul E. McKenney
The underlying problem is that perf is invoking
call_rcu()with the scheduler locks held, but in NOCB mode,call_rcu()will with high probability invoke the scheduler -- which just might want to use its locks. The reason thatcall_rcu()needs to invoke the scheduler is to wake up the corresponding rcuo callback-offload kthread, which does the job of starting up a grace period and invoking the callbacks afterwards.One solution (championed on a related problem by Lai Jiangshan) is to simply defer the wakeup to some point where scheduler locks are no longer held. Since we don't want to unnecessarily incur the cost of such deferral, the task before us is threefold:
1. Determine when it is likely that a relevant scheduler lock is held.
2. Defer the wakeup in such cases.
3. Ensure that all deferred wakeups eventually happen, preferably
sooner rather than later.We use
irqs_disabled_flags()as a proxy for relevant scheduler locks being held. This works because the relevant locks are always acquired with interrupts disabled. We may defer more often than needed, but that is at least safe.
Quoted 6 lines
Quoted 2 lines
This would also allow us to do away with things like the below patch, right?
It takes care of one problem, but there are others, including
rcu_read_unlock()inovking the scheduler to deboost itself. So for the moment, we still need the below patch.
Oh right, see I knew I was forgetting something... :-)
Quoted 28 lines · Peter Zijlstra
Quoted 27 lines · Paul E. McKenney
Quoted 26 lines · Peter Zijlstra
Quoted 24 lines · tip-bot for Paul E. McKenney
The underlying problem is that perf is invoking
call_rcu()with the scheduler locks held, but in NOCB mode,call_rcu()will with high probability invoke the scheduler -- which just might want to use its locks. The reason thatcall_rcu()needs to invoke the scheduler is to wake up the corresponding rcuo callback-offload kthread, which does the job of starting up a grace period and invoking the callbacks afterwards.One solution (championed on a related problem by Lai Jiangshan) is to simply defer the wakeup to some point where scheduler locks are no longer held. Since we don't want to unnecessarily incur the cost of such deferral, the task before us is threefold:
1. Determine when it is likely that a relevant scheduler lock is held.
2. Defer the wakeup in such cases.
3. Ensure that all deferred wakeups eventually happen, preferably
sooner rather than later.We use
irqs_disabled_flags()as a proxy for relevant scheduler locks being held. This works because the relevant locks are always acquired with interrupts disabled. We may defer more often than needed, but that is at least safe.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
This would also allow us to do away with things like the below patch, right?
It takes care of one problem, but there are others, including
rcu_read_unlock()inovking the scheduler to deboost itself. So for the moment, we still need the below patch.Oh right, see I knew I was forgetting something... :-)
I am hoping to make your patch unnecessary, but it ain't trivial. ;-)
We will get there! Especially if I can find Lai Jiangshan's old patch that reworked deboosting. :-/
Thanx, Paul