xfs: insert memory barriers before wake_up_bit()
cover xfs: insert memory barriers before wake_up_bit()
These two patches insert memory barriers in xfs_iunlock() and xfs_inode_item_unpin() before their calls to wake_up_bit(), to ensure just-queued processes are properly awakened.
-Alex
[PATCH 1/2] xfs: memory barrier before wake_up_bit() [PATCH 2/2] xfs: another memory barrier before wake_up_bit()
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/2 xfs: memory barrier before wake_up_bit()
In xfs_ifunlock() there is a call to wake_up_bit() after clearing the flush lock on the xfs inode. This is not guaranteed to be safe, as noted in the comments above wake_up_bit() beginning with:
In order for this to function properly, as it uses waitqueue_active() internally, some kind of memory barrier must be done prior to calling this.
I claim no mastery of the details and subtlety of memory barrier use, but I believe the issue is that the call to waitqueue_active() in __wake_up_bit(), could be operating on a value of "wq" that is out of date. This patch fixes this by inserting a call to smp_mb() in xfs_iunlock before calling wake_up_bit(), along the lines of what's done in unlock_new_inode(). A litte more explanation follows.
In __xfs_iflock(), prepare_to_wait_exclusive() adds a wait queue entry to the end of a bit wait queue before setting the current task state to UNINTERRUPTIBLE. And although setting the task state issues a full smp_mb() (which ensures changes made are visible to the rest of the system at that point) that alone does not guarantee that other CPUs will instantly avail themselves of the updated value. A separate CPU needs to issue at least a read barrier in order to ensure the wq value it uses to determine whether there are waiters is up-to-date, and waitqueue_active() does not do that.
I came to suspect this code because we had a customer with a system that was hung with one or more tasks stuck in __xfs_iflock(). A little poking around the affected code led me to the comments in wake_up_bit().
--- fs/xfs/xfs_inode.h | 1 + 1 file changed, 1 insertion(+)
fs/xfs/xfs_inode.h +1 -0
@@ -419,6 +419,7 @@ static inline void xfs_iflock(struct xfs_inode *ip)static inline void xfs_ifunlock (struct xfs_inode *ip) {xfs_iflags_clear (ip,XFS_IFLOCK ); +smp_mb ();wake_up_bit (&ip->i_flags, __XFS_IFLOCK_BIT); }
Signature · 6 lines
1.7.9.5 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/2 xfs: another memory barrier before wake_up_bit()
In reply to Alex ElderIn xfs_inode_item_unpin() there is a call to wake_up_bit() following an independent test for whether waiters should be awakened. This requires a memory barrier in order to guarantee correct operation (see the comment above wake_up_bit()).
--- fs/xfs/xfs_inode_item.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
fs/xfs/xfs_inode_item.c +4 -2
@@ -474,8 +474,10 @@ xfs_inode_item_unpin(trace_xfs_inode_unpin (ip, _RET_IP_);ASSERT (atomic_read (&ip->i_pincount) >0 ); -if (atomic_dec_and_test (&ip->i_pincount)) -wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT); +if (!atomic_dec_and_test (&ip->i_pincount)) +return ; +smp_mb (); +wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT); }STATIC uint
Signature · 6 lines
1.7.9.5 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/2] xfs: memory barrier before wake_up_bit()
In reply to Alex ElderQuoted 17 lines · Alex Elder
In
xfs_ifunlock()there is a call towake_up_bit()after clearing the flush lock on the xfs inode. This is not guaranteed to be safe, as noted in the comments abovewake_up_bit()beginning with:In order for this to function properly, as it uses
waitqueue_active()internally, some kind of memory barrier must be done prior to calling this.I claim no mastery of the details and subtlety of memory barrier use, but I believe the issue is that the call to
waitqueue_active()in__wake_up_bit(), could be operating on a value of "wq" that is out of date. This patch fixes this by inserting a call tosmp_mb()inxfs_iunlockbefore callingwake_up_bit(), along the lines of what's done inunlock_new_inode(). A litte more explanation follows.
Quoted 9 lines
In
__xfs_iflock(),prepare_to_wait_exclusive()adds a wait queue entry to the end of a bit wait queue before setting the current task state to UNINTERRUPTIBLE. And although setting the task state issues a fullsmp_mb()(which ensures changes made are visible to the rest of the system at that point) that alone does not guarantee that other CPUs will instantly avail themselves of the updated value. A separate CPU needs to issue at least a read barrier in order to ensure the wq value it uses to determine whether there are waiters is up-to-date, andwaitqueue_active()does not do that.
You can probably trim most of this and simply point at the comment describing wake_up_bit()....
Quoted 15 lines
I came to suspect this code because we had a customer with a system that was hung with one or more tasks stuck in
__xfs_iflock(). A little poking around the affected code led me to the comments inwake_up_bit().Signed-off-by Alex Elder <[email protected]>--- fs/xfs/xfs_inode.h | 1 + 1 file changed, 1 insertion(+)fs/xfs/xfs_inode.h +0
-0
fs/xfs/xfs_inode.h static inline void xfs_iflock(struct xfs_inode *ip) · line 423
Quoted 5 lines
fs/xfs/xfs_inode.h +1
-0@@ -419,6 +419,7 @@ static inline void xfs_iflock(struct xfs_inode *ip)static inline void xfs_ifunlock (struct xfs_inode *ip) {xfs_iflags_clear (ip,XFS_IFLOCK ); +smp_mb ();wake_up_bit (&ip->i_flags, __XFS_IFLOCK_BIT);
ACK, smp_mb() is needed because spin_unlock() is not a memory barrier and so not everyone will have seen the bit being cleared.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/2] xfs: another memory barrier before wake_up_bit()
In reply to Alex ElderQuoted 15 lines · Alex Elder
In
xfs_inode_item_unpin()there is a call towake_up_bit()following an independent test for whether waiters should be awakened. This requires a memory barrier in order to guarantee correct operation (see the comment abovewake_up_bit()).Signed-off-by Alex Elder <[email protected]>--- fs/xfs/xfs_inode_item.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fs/xfs/xfs_inode_item.c +0
-0
fs/xfs/xfs_inode_item.c xfs_inode_item_unpin( · line 480
Quoted 9 lines
fs/xfs/xfs_inode_item.c +4
-2@@ -474,8 +474,10 @@ xfs_inode_item_unpin(trace_xfs_inode_unpin (ip, _RET_IP_);ASSERT (atomic_read (&ip->i_pincount) >0 ); -if (atomic_dec_and_test (&ip->i_pincount)) -wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT); +if (!atomic_dec_and_test (&ip->i_pincount)) +return ; +smp_mb (); +wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT);
I'm not sure this a barrier is actually needed here. The "wake up" bit is never stored or cleared anywhere in this case, it is used only to define a wait channel and directed wake up. Hence the "need a barrier so all CPUs see the cleared bit" case doesn't arise here. We use an atomic variable instead, and that makes it safe.
If you read Documentation/atomic_ops.txt, you'll find that atomic modification operations are required to have explicit barrier semantics. i.e. that atomic_dec_and_test() must behave like it has both a smp_mb() before and after the atomic operation. i.e:
Unlike the above routines, it is required that explicit memory barriers are performed before and after the operation. It must be done such that all memory operations before and after the atomic operation calls are strongly ordered with respect to the atomic operation itself.
So, the smp_mb() that is added here is redundant - the atomic_dec_and_test() call already has the necesary memory barriers that wake_up_bit() requires.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/2] xfs: memory barrier before wake_up_bit()
In reply to Dave ChinnerQuoted 28 lines · Dave Chinner
Quoted 17 lines · Alex Elder
In
xfs_ifunlock()there is a call towake_up_bit()after clearing the flush lock on the xfs inode. This is not guaranteed to be safe, as noted in the comments abovewake_up_bit()beginning with:In order for this to function properly, as it uses
waitqueue_active()internally, some kind of memory barrier must be done prior to calling this.I claim no mastery of the details and subtlety of memory barrier use, but I believe the issue is that the call to
waitqueue_active()in__wake_up_bit(), could be operating on a value of "wq" that is out of date. This patch fixes this by inserting a call tosmp_mb()inxfs_iunlockbefore callingwake_up_bit(), along the lines of what's done inunlock_new_inode(). A litte more explanation follows.Quoted 9 lines
In
__xfs_iflock(),prepare_to_wait_exclusive()adds a wait queue entry to the end of a bit wait queue before setting the current task state to UNINTERRUPTIBLE. And although setting the task state issues a fullsmp_mb()(which ensures changes made are visible to the rest of the system at that point) that alone does not guarantee that other CPUs will instantly avail themselves of the updated value. A separate CPU needs to issue at least a read barrier in order to ensure the wq value it uses to determine whether there are waiters is up-to-date, andwaitqueue_active()does not do that.
Quoted 2 lines
You can probably trim most of this and simply point at the comment describing
wake_up_bit()....
Yeah, I know. I just wanted to sort of say what I was thinking to get confirmation (or correction). I now have a much better understanding of barriers than I did before, but there are still corners I haven't wrapped my head around.
Ben, please feel free do trim off this stuff as you see fit.
-Alex
Quoted 25 lines
Quoted 16 lines
I came to suspect this code because we had a customer with a system that was hung with one or more tasks stuck in
__xfs_iflock(). A little poking around the affected code led me to the comments inwake_up_bit().Signed-off-by Alex Elder <[email protected]>--- fs/xfs/xfs_inode.h | 1 + 1 file changed, 1 insertion(+)fs/xfs/xfs_inode.h +0
-0fs/xfs/xfs_inode.h static inline void xfs_iflock(struct xfs_inode *ip) · line 423
Quoted 5 lines
fs/xfs/xfs_inode.h +1
-0@@ -419,6 +419,7 @@ static inline void xfs_iflock(struct xfs_inode *ip)static inline void xfs_ifunlock (struct xfs_inode *ip) {xfs_iflags_clear (ip,XFS_IFLOCK ); +smp_mb ();wake_up_bit (&ip->i_flags, __XFS_IFLOCK_BIT);ACK,
smp_mb()is needed becausespin_unlock()is not a memory barrier and so not everyone will have seen the bit being cleared.
Quoted 6 lines
Reviewed-by Dave Chinner <[email protected]>Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/2] xfs: another memory barrier before wake_up_bit()
In reply to Dave ChinnerQuoted 43 lines · Dave Chinner
Quoted 15 lines · Alex Elder
In
xfs_inode_item_unpin()there is a call towake_up_bit()following an independent test for whether waiters should be awakened. This requires a memory barrier in order to guarantee correct operation (see the comment abovewake_up_bit()).Signed-off-by Alex Elder <[email protected]>--- fs/xfs/xfs_inode_item.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)fs/xfs/xfs_inode_item.c +0
-0fs/xfs/xfs_inode_item.c xfs_inode_item_unpin( · line 480
Quoted 9 lines
fs/xfs/xfs_inode_item.c +4
-2@@ -474,8 +474,10 @@ xfs_inode_item_unpin(trace_xfs_inode_unpin (ip, _RET_IP_);ASSERT (atomic_read (&ip->i_pincount) >0 ); -if (atomic_dec_and_test (&ip->i_pincount)) -wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT); +if (!atomic_dec_and_test (&ip->i_pincount)) +return ; +smp_mb (); +wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT);I'm not sure this a barrier is actually needed here. The "wake up" bit is never stored or cleared anywhere in this case, it is used only to define a wait channel and directed wake up. Hence the "need a barrier so all CPUs see the cleared bit" case doesn't arise here. We use an atomic variable instead, and that makes it safe.
If you read
Documentation/atomic_ops.txt, you'll find that atomic modification operations are required to have explicit barrier semantics. i.e. thatatomic_dec_and_test()must behave like it has both asmp_mb()before and after the atomic operation. i.e:Unlike the above routines, it is required that explicit memory barriers are performed before and after the operation. It must be done such that all memory operations before and after the atomic operation calls are strongly ordered with respect to the atomic operation itself.
Quoted 3 lines
So, the
smp_mb()that is added here is redundant - theatomic_dec_and_test()call already has the necesary memory barriers thatwake_up_bit()requires.
I hadn't looked at that in as much detail, but now that you point it out I concur.
I retract this patch.
Thanks.
-Alex
Quoted 5 lines
Cheers,
Dave.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Ben Myers
AppliedRe: [PATCH 1/2] xfs: memory barrier before wake_up_bit()
In reply to Alex ElderQuoted 32 lines · Alex Elder
Quoted 28 lines · Dave Chinner
Quoted 17 lines · Alex Elder
In
xfs_ifunlock()there is a call towake_up_bit()after clearing the flush lock on the xfs inode. This is not guaranteed to be safe, as noted in the comments abovewake_up_bit()beginning with:In order for this to function properly, as it uses
waitqueue_active()internally, some kind of memory barrier must be done prior to calling this.I claim no mastery of the details and subtlety of memory barrier use, but I believe the issue is that the call to
waitqueue_active()in__wake_up_bit(), could be operating on a value of "wq" that is out of date. This patch fixes this by inserting a call tosmp_mb()inxfs_iunlockbefore callingwake_up_bit(), along the lines of what's done inunlock_new_inode(). A litte more explanation follows.Quoted 9 lines
In
__xfs_iflock(),prepare_to_wait_exclusive()adds a wait queue entry to the end of a bit wait queue before setting the current task state to UNINTERRUPTIBLE. And although setting the task state issues a fullsmp_mb()(which ensures changes made are visible to the rest of the system at that point) that alone does not guarantee that other CPUs will instantly avail themselves of the updated value. A separate CPU needs to issue at least a read barrier in order to ensure the wq value it uses to determine whether there are waiters is up-to-date, andwaitqueue_active()does not do that.Quoted 2 lines
You can probably trim most of this and simply point at the comment describing
wake_up_bit()....
Quoted 8 lines
Yeah, I know. I just wanted to sort of say what I was thinking to get confirmation (or correction). I now have a much better understanding of barriers than I did before, but there are still corners I haven't wrapped my head around.
Ben, please feel free do trim off this stuff as you see fit.
Applied.
Thanks Alex!
-Ben
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs