xfs: fix missed wakeup on l_flush_wait
PATCH 1/1 xfs: fix missed wakeup on l_flush_wait
From: Rik van Riel <[email protected]>
commit cdea5459ce263fbc963657a7736762ae897a8ae6 upstream
The code in xlog_wait uses the spinlock to make adding the task to the wait queue, and setting the task state to UNINTERRUPTIBLE atomic with respect to the waker.
Doing the wakeup after releasing the spinlock opens up the following race condition:
Task 1 task 2
add task to wait queue
wake up task
set task state to UNINTERRUPTIBLE
This issue was found through code inspection as a result of kworkers being observed stuck in UNINTERRUPTIBLE state with an empty wait queue. It is rare and largely unreproducable.
Simply moving the spin_unlock to after the wake_up_all results in the waker not being able to see a task on the waitqueue before it has set its state to UNINTERRUPTIBLE.
This bug dates back to the conversion of this code to generic waitqueue infrastructure from a counting semaphore back in 2008 which didn't place the wakeups consistently w.r.t. to the relevant spin locks.
[dchinner: Also fix a similar issue in the shutdown path on xc_commit_wait. Update commit log with more details of the issue.]
---
This issue was fixed in v5.4 but didn't appear to make it to stable. The fixed commit goes back to v2.6, so backport this to stable kernels before v5.4. The only difference is a contextual change at
xlog_state_do_callback(log, XFS_LI_ABORTED, NULL); Which in v5.4 is xlog_state_do_callback(log, true, NULL);
fs/xfs/xfs_log.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
fs/xfs/xfs_log.c +4 -5
@@ -2684,7 +2684,6 @@ xlog_state_do_callback(int funcdidcallbacks;/* flag: function did callbacks */ int repeats;/* for issuing console warnings if * looping too many times */ -int wake =0 ;spin_lock (&log->l_icloglock); first_iclog = iclog = log->l_iclog;
@@ -2886,11 +2885,9 @@ xlog_state_do_callback(#endif if (log->l_iclog->ic_state & (XLOG_STATE_ACTIVE |XLOG_STATE_IOERROR )) - wake =1 ; -spin_unlock (&log->l_icloglock); - -if (wake)wake_up_all (&log->l_flush_wait); + +spin_unlock (&log->l_icloglock); }
@@ -4052,7 +4049,9 @@ xfs_log_force_umount(* item committed callback functions will do this again under lock to * avoid races. */ +spin_lock (&log->l_cilp->xc_push_lock);wake_up_all (&log->l_cilp->xc_commit_wait); +spin_unlock (&log->l_cilp->xc_push_lock);xlog_state_do_callback (log,XFS_LI_ABORTED ,NULL );#ifdef XFSERRORDEBUG
Signature · 1 line
2.17.1
Quoted 45 lines · Samuel Mendoza-Jonas
From: Rik van Riel <[email protected]>
commit
cdea5459ce263fbc963657a7736762ae897a8ae6upstreamThe code in
xlog_waituses the spinlock to make adding the task to the wait queue, and setting the task state to UNINTERRUPTIBLE atomic with respect to the waker.Doing the wakeup after releasing the spinlock opens up the following race condition:
Task 1 task 2
add task to wait queue
wake up task
set task state to UNINTERRUPTIBLEThis issue was found through code inspection as a result of kworkers being observed stuck in UNINTERRUPTIBLE state with an empty wait queue. It is rare and largely unreproducable.
Simply moving the
spin_unlockto after thewake_up_allresults in the waker not being able to see a task on the waitqueue before it has set its state to UNINTERRUPTIBLE.This bug dates back to the conversion of this code to generic waitqueue infrastructure from a counting semaphore back in 2008 which didn't place the wakeups consistently w.r.t. to the relevant spin locks.
[dchinner: Also fix a similar issue in the shutdown path on
xc_commit_wait. Update commit log with more details of the issue.]Fixes d748c62367eb ("[XFS] Convert l_flushsema to a sv_t")Reported-by Chris Mason <[email protected]>Signed-off-by Rik van Riel <[email protected]>Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Darrick J. Wong <[email protected]>Cc [email protected] # 4.9.x-4.19.x[modified for contextual change near xlog_state_do_callback()]Signed-off-by Samuel Mendoza-Jonas <[email protected]>Reviewed-by Frank van der Linden <[email protected]>Reviewed-by Suraj Jitindar Singh <[email protected]>Reviewed-by Benjamin Herrenschmidt <[email protected]>Reviewed-by Anchal Agarwal <[email protected]>
Quoted 7 lines
---
This issue was fixed in v5.4 but didn't appear to make it to stable. The fixed commit goes back to v2.6, so backport this to stable kernels before v5.4. The only difference is a contextual change atxlog_state_do_callback(log, XFS_LI_ABORTED, NULL); Which in v5.4 is xlog_state_do_callback(log, true, NULL);
Now queued up, thanks.
greg k-h