fix lock inversion in lockless sys_times()
Subsystems (2)
cover fix lock inversion in lockless sys_times()
The sig->stats_lock nests inside the tasklist_lock and the sighand->siglock in __exit_signal and wait_task_zombie.
However, both of those locks can be taken from irq context, which means we need to use the interrupt safe variant of read_seqbegin_or_lock. This blocks interrupts when the "lock" branch is taken (seq is odd), preventing the lock inversion.
On the first (lockless) pass through the loop, irqs are not blocked.
This fixes the lockdep complaints that Stanislaw reported.
PATCH 1/2 seqlock: add irqsave variant of read_seqbegin_or_lock
From: Rik van Riel <[email protected]>
There are cases where read_seqbegin_or_lock needs to block irqs, because the seqlock in question nests inside a lock that is also be taken from irq context.
Add read_seqbegin_or_lock_irqsave and done_seqretry_irqrestore, which are almost identical to read_seqbegin_or_lock and done_seqretry.
--- include/linux/seqlock.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
include/linux/seqlock.h +17 -0
@@ -456,4 +456,21 @@ read_sequnlock_excl_irqrestore(seqlock_t *sl, unsigned long flags) spin_unlock_irqrestore(&sl->lock, flags); } +static inline unsigned long read_seqbegin_or_lock_irqsave (seqlock_t *lock, +int *seq) +{ +unsigned long flags =0 ; +if (!(*seq &1 ))/* Even */ + *seq =read_seqbegin (lock); +else /* Odd */ +read_seqlock_excl_irqsave (lock, flags); +return flags; +} + +static inline void done_seqretry_irqrestore (seqlock_t *lock,int seq, +unsigned long flags) +{ +if (seq &1 ) +read_sequnlock_excl_irqrestore (lock, flags); +}#endif /* __LINUX_SEQLOCK_H */
Signature · 1 line
1.9.3
tip-bot for Rik van Riel
merge bot[tip:sched/core] seqlock: Add irqsave variant of read_seqbegin_or_lock()
seqlock: Add irqsave variant of read_seqbegin_or_lock()
Commit-ID: ef8ac06359ddf95431cf6bb04ad2b36fff562328 Gitweb: http://git.kernel.org/tip/ef8ac06359ddf95431cf6bb04ad2b36fff562328 Author: Rik van Riel <[email protected]> AuthorDate: Fri, 12 Sep 2014 09:12:14 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Sep 2014 12:35:16 +0200
seqlock: Add irqsave variant of read_seqbegin_or_lock()
There are cases where read_seqbegin_or_lock() needs to block irqs, because the seqlock in question nests inside a lock that is also be taken from irq context.
Add read_seqbegin_or_lock_irqsave() and done_seqretry_irqrestore(), which are almost identical to read_seqbegin_or_lock() and done_seqretry().
--- include/linux/seqlock.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
include/linux/seqlock.h +19 -0
@@ -456,4 +456,23 @@ read_sequnlock_excl_irqrestore(seqlock_t *sl, unsigned long flags) spin_unlock_irqrestore(&sl->lock, flags); } +static inline unsigned long +read_seqbegin_or_lock_irqsave (seqlock_t *lock,int *seq) +{ +unsigned long flags =0 ; + +if (!(*seq &1 ))/* Even */ + *seq =read_seqbegin (lock); +else /* Odd */ +read_seqlock_excl_irqsave (lock, flags); + +return flags; +} + +static inline void +done_seqretry_irqrestore (seqlock_t *lock,int seq,unsigned long flags) +{ +if (seq &1 ) +read_sequnlock_excl_irqrestore (lock, flags); +}#endif /* __LINUX_SEQLOCK_H */
PATCH 2/2 sched,time: fix lock inversion in thread_group_cputime
In reply to [email protected]From: Rik van Riel <[email protected]>
The sig->stats_lock nests inside the tasklist_lock and the sighand->siglock in __exit_signal and wait_task_zombie.
However, both of those locks can be taken from irq context, which means we need to use the interrupt safe variant of read_seqbegin_or_lock. This blocks interrupts when the "lock" branch is taken (seq is odd), preventing the lock inversion.
On the first (lockless) pass through the loop, irqs are not blocked.
--- kernel/sched/cputime.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/cputime.c +3 -2
@@ -289,13 +289,14 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)cputime_t utime, stime;struct task_struct *t;unsigned int seq, nextseq; +unsigned long flags;rcu_read_lock ();/* Attempt a lockless read on the first round. */ nextseq =0 ; do { seq = nextseq; - read_seqbegin_or_lock(&sig->stats_lock, &seq); + flags = read_seqbegin_or_lock_irqsave(&sig->stats_lock, &seq); times->utime = sig->utime; times->stime = sig->stime; times->sum_exec_runtime = sig->sum_sched_runtime;
@@ -309,7 +310,7 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)/* If lockless access failed, take the lock. */ nextseq =1 ; }while (need_seqretry (&sig->stats_lock, seq)); -done_seqretry (&sig->stats_lock, seq); +done_seqretry_irqrestore (&sig->stats_lock, seq, flags);rcu_read_unlock (); }
Signature · 1 line
1.9.3
tip-bot for Rik van Riel
merge bot[tip:sched/core] sched, time: Fix lock inversion in thread_group_cputime()
sched, time: Fix lock inversion in thread_group_cputime()
Commit-ID: 9c368b5b6eccce1cbd7f68142106b3b4ddb1c5b5 Gitweb: http://git.kernel.org/tip/9c368b5b6eccce1cbd7f68142106b3b4ddb1c5b5 Author: Rik van Riel <[email protected]> AuthorDate: Fri, 12 Sep 2014 09:12:15 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Fri, 19 Sep 2014 12:35:17 +0200
sched, time: Fix lock inversion in thread_group_cputime()
The sig->stats_lock nests inside the tasklist_lock and the sighand->siglock in __exit_signal and wait_task_zombie.
However, both of those locks can be taken from irq context, which means we need to use the interrupt safe variant of read_seqbegin_or_lock. This blocks interrupts when the "lock" branch is taken (seq is odd), preventing the lock inversion.
On the first (lockless) pass through the loop, irqs are not blocked.
--- kernel/sched/cputime.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/sched/cputime.c +3 -2
@@ -289,13 +289,14 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)cputime_t utime, stime;struct task_struct *t;unsigned int seq, nextseq; +unsigned long flags;rcu_read_lock ();/* Attempt a lockless read on the first round. */ nextseq =0 ; do { seq = nextseq; - read_seqbegin_or_lock(&sig->stats_lock, &seq); + flags = read_seqbegin_or_lock_irqsave(&sig->stats_lock, &seq); times->utime = sig->utime; times->stime = sig->stime; times->sum_exec_runtime = sig->sum_sched_runtime;
@@ -309,7 +310,7 @@ void thread_group_cputime(struct task_struct *tsk, struct task_cputime *times)/* If lockless access failed, take the lock. */ nextseq =1 ; }while (need_seqretry (&sig->stats_lock, seq)); -done_seqretry (&sig->stats_lock, seq); +done_seqretry_irqrestore (&sig->stats_lock, seq, flags);rcu_read_unlock (); }
Peter Zijlstra
maintainerQuoted 8 lines · On Fri, Sep 12, 2014 at 09:12:13AM -0400, [email protected] wrote:
The
sig->stats_locknests inside thetasklist_lockand thesighand->siglockin__exit_signalandwait_task_zombie.However, both of those locks can be taken from irq context, which means we need to use the interrupt safe variant of
read_seqbegin_or_lock. This blocks interrupts when the "lock" branch is taken (seq is odd), preventing the lock inversion.
Quoted 5 lines
On the first (lockless) pass through the loop, irqs are not blocked.
This fixes the lockdep complaints that Stanislaw reported.
Thanks Rik