rwsem: fix rwsem_is_locked() bugs
Subsystems (2)
PATCH v4 1/1 rwsem: fix rwsem_is_locked() bugs
rwsem_is_locked() tests ->activity without locks, so we should always keep ->activity consistent. However, the code in __rwsem_do_wake() breaks this rule, it updates ->activity after _all_ readers waken up, this may give some reader a wrong ->activity value, thus cause rwsem_is_locked() behaves wrong.
Quote from Andrew:
"
- we have one or more processes sleeping in down_read(), waiting for access.
- we wake one or more processes up without altering ->activity
- they start to run and they do rwsem_is_locked(). This incorrectly returns "false", because the waker process is still crunching away in __rwsem_do_wake().
- the waker now alters ->activity, but it was too late. "
So we need get a spinlock to protect this. And rwsem_is_locked() should not block, thus we use spin_trylock.
---
include/linux/rwsem-spinlock.h +7 -1
@@ -71,7 +71,13 @@ extern void __downgrade_write(struct rw_semaphore *sem);static inline int rwsem_is_locked (struct rw_semaphore *sem) { -return (sem->activity !=0 ); +int ret =1 ; + +if (spin_trylock_irq (&sem->wait_lock)) { + ret = (sem->activity !=0 ); +spin_unlock_irq (&sem->wait_lock); + } +return ret; }#endif /* __KERNEL__ */
lib/rwsem-spinlock.c +4 -0
@@ -82,6 +82,10 @@ __rwsem_do_wake(struct rw_semaphore *sem, int wakewrite)while (waiter->flags &RWSEM_WAITING_FOR_READ ) {struct list_head *next = waiter->list.next; +/* +* Since rwsem_is_locked() reads ->activity with spinlock, +* not updating ->activity here is fine. +*/ list_del (&waiter->list); tsk = waiter->task;smp_mb ();
Quoted 20 lines · Amerigo Wang
rwsem_is_locked()tests ->activity without locks, so we should always keep ->activity consistent. However, the code in__rwsem_do_wake()breaks this rule, it updates ->activity after _all_ readers waken up, this may give some reader a wrong ->activity value, thus causerwsem_is_locked()behaves wrong.Quote from Andrew:
"
- we have one or more processes sleeping indown_read(), waiting for access.- we wake one or more processes up without altering ->activity
- they start to run and they do
rwsem_is_locked(). This incorrectly returns "false", because the waker process is still crunching away in__rwsem_do_wake().- the waker now alters ->activity, but it was too late. "
Quoted 7 lines
So we need get a spinlock to protect this. And
rwsem_is_locked()should not block, thus we usespin_trylock.Reported-by Brian Behlendorf <[email protected]>Cc Ben Woodard <[email protected]>Cc David Howells <[email protected]>Signed-off-by WANG Cong <[email protected]>
I'd say the comment in __rwsem_do_wake() is unnecessary, but other than that...
Quoted 25 lines · David Howells
Quoted 23 lines · Amerigo Wang
rwsem_is_locked()tests ->activity without locks, so we should always keep ->activity consistent. However, the code in__rwsem_do_wake()breaks this rule, it updates ->activity after _all_ readers waken up, this may give some reader a wrong ->activity value, thus causerwsem_is_locked()behaves wrong.Quote from Andrew:
"
- we have one or more processes sleeping indown_read(), waiting for access.- we wake one or more processes up without altering ->activity
- they start to run and they do
rwsem_is_locked(). This incorrectly returns "false", because the waker process is still crunching away in__rwsem_do_wake().- the waker now alters ->activity, but it was too late. "
So we need get a spinlock to protect this. And
rwsem_is_locked()should not block, thus we usespin_trylock.
Quoted 7 lines
Quoted 4 lines
Reported-by Brian Behlendorf <[email protected]>Cc Ben Woodard <[email protected]>Cc David Howells <[email protected]>Signed-off-by WANG Cong <[email protected]>I'd say the comment in
__rwsem_do_wake()is unnecessary, but other than that...
The reason why I added it is to show that we have considered that case already. :) If you have strong opinions to remove it, I can update the patch.
Quoted 2 lines
Acked-by David Howells <[email protected]>
Thanks!
Andrew Morton
maintainerQuoted 5 lines · On Thu, 8 Oct 2009 05:23:53 -0400 Amerigo Wang <[email protected]> wrote:
include/linux/rwsem-spinlock.h +0
-0@@ -71,7 +71,13 @@ extern void __downgrade_write(struct rw_semaphore *sem);static inline int rwsem_is_locked (struct rw_semaphore *sem)
include/linux/rwsem-spinlock.h extern void __downgrade_write(struct rw_semaphore *sem); · line 81
Quoted 10 lines
include/linux/rwsem-spinlock.h +7
-1{ -return (sem->activity !=0 ); +int ret =1 ; + +if (spin_trylock_irq (&sem->wait_lock)) { + ret = (sem->activity !=0 ); +spin_unlock_irq (&sem->wait_lock); + } +return ret; }
a) probably to large to be inlined
b) the function will now cause bugs if called under local_irq_disable(). That wasn't the case before. Fixable via spin_lock_irqsave().
In the present kernel there don't appear to be any irqs-off callers. There may of course be some out-of-tree ones which will get bitten by this semantic change.
If we decide to leave this new rule in place then we should add a WARN_ON(irqs_disabled()) to prevent hitting people with a nasty, subtle bug.
Methinks that _irqsave() is better.
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Thu, 8 Oct 2009 05:23:53 -0400 Amerigo Wang <[email protected]> wrote:
include/linux/rwsem-spinlock.h +0
-0@@ -71,7 +71,13 @@ extern void __downgrade_write(struct rw_semaphore *sem);
Quoted 13 lines
include/linux/rwsem-spinlock.h extern void __downgrade_write(struct rw_semaphore *sem); · line 81
Quoted 11 lines
include/linux/rwsem-spinlock.h +7
-1static inline int rwsem_is_locked (struct rw_semaphore *sem) { -return (sem->activity !=0 ); +int ret =1 ; + +if (spin_trylock_irq (&sem->wait_lock)) { + ret = (sem->activity !=0 ); +spin_unlock_irq (&sem->wait_lock); + } +return ret; }a) probably to large to be inlined
Yeah, maybe, I forgot spin_trylock_irq() and spin_unlock_irq are macros.
Quoted 5 lines
b) the function will now cause bugs if called under
local_irq_disable(). That wasn't the case before. Fixable viaspin_lock_irqsave().
Quoted 8 lines
In the present kernel there don't appear to be any irqs-off callers. There may of course be some out-of-tree ones which will get bitten by this semantic change.
If we decide to leave this new rule in place then we should add a WARN_ON(irqs_disabled()) to prevent hitting people with a nasty, subtle bug.
Methinks that
_irqsave()is better.
My bad, I misunderstood spin_lock_irqsave(), thus used spin_lock_irq(). :( Will update the patch now.
Thanks!