xfs: end sync buffer I/O properly on shutdown error
PATCH 1/1 xfs: end sync buffer I/O properly on shutdown error
As of commit e339dd8d8b ("xfs: use sync buffer I/O for sync delwri queue submission"), the delwri submission code uses sync buffer I/O for sync delwri I/O. Instead of waiting on async I/O to unlock the buffer, it uses the underlying sync I/O completion mechanism.
If delwri buffer submission fails due to a shutdown scenario, an error is set on the buffer and buffer completion never occurs. This can cause xfs_buf_delwri_submit() to deadlock waiting on a completion event.
We could check the error state before waiting on such buffers, but that doesn't serialize against the case of an error set via a racing I/O completion. Instead, invoke I/O completion in the shutdown case regardless of buffer I/O type.
--- fs/xfs/xfs_buf.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
fs/xfs/xfs_buf.c +1 -2
@@ -1536,8 +1536,7 @@ __xfs_buf_submit(xfs_buf_ioerror (bp, -EIO ); bp->b_flags &= ~XBF_DONE ;xfs_buf_stale (bp); -if (bp->b_flags &XBF_ASYNC ) -xfs_buf_ioend (bp); +xfs_buf_ioend (bp);return -EIO ; }
Signature · 1 line
2.17.2
Quoted 10 lines · Brian Foster
As of commit
e339dd8d8b("xfs: use sync buffer I/O for sync delwri queue submission"), the delwri submission code uses sync buffer I/O for sync delwri I/O. Instead of waiting on async I/O to unlock the buffer, it uses the underlying sync I/O completion mechanism.If delwri buffer submission fails due to a shutdown scenario, an error is set on the buffer and buffer completion never occurs. This can cause
xfs_buf_delwri_submit()to deadlock waiting on a completion event.
Quoted 4 lines
We could check the error state before waiting on such buffers, but that doesn't serialize against the case of an error set via a racing I/O completion. Instead, invoke I/O completion in the shutdown case regardless of buffer I/O type.
How did you find this? i.e. what are the symptoms of the bug? I'm guessing that it's a shutdown/unmount hang from the above, but I'm really not sure.
Quoted 10 lines
Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_buf.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)fs/xfs/xfs_buf.c +0
-0
fs/xfs/xfs_buf.c __xfs_buf_submit( · line 1541
Quoted 8 lines
fs/xfs/xfs_buf.c +1
-2@@ -1536,8 +1536,7 @@ __xfs_buf_submit(xfs_buf_ioerror (bp, -EIO ); bp->b_flags &= ~XBF_DONE ;xfs_buf_stale (bp); -if (bp->b_flags &XBF_ASYNC ) -xfs_buf_ioend (bp); +xfs_buf_ioend (bp);return -EIO ; }
That said, it definitely looks like it fixes a bug. Will test.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 11 lines · Dave Chinner
Quoted 10 lines · Brian Foster
As of commit
e339dd8d8b("xfs: use sync buffer I/O for sync delwri queue submission"), the delwri submission code uses sync buffer I/O for sync delwri I/O. Instead of waiting on async I/O to unlock the buffer, it uses the underlying sync I/O completion mechanism.If delwri buffer submission fails due to a shutdown scenario, an error is set on the buffer and buffer completion never occurs. This can cause
xfs_buf_delwri_submit()to deadlock waiting on a completion event.
Quoted 9 lines
Quoted 4 lines
We could check the error state before waiting on such buffers, but that doesn't serialize against the case of an error set via a racing I/O completion. Instead, invoke I/O completion in the shutdown case regardless of buffer I/O type.
How did you find this? i.e. what are the symptoms of the bug? I'm guessing that it's a shutdown/unmount hang from the above, but I'm really not sure.
A shutdown during log recovery via generic/034 reproduced the deadlock described in the commit log. The shutdown itself was caused by developer error (missing an xfs_buf_ops assignment when working on the magic stuff), so I'm not aware of a current upstream reproducer (that wouldn't be related to some already corrupted fs).
Quoted 21 lines
Quoted 10 lines
Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_buf.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)fs/xfs/xfs_buf.c +0
-0fs/xfs/xfs_buf.c __xfs_buf_submit( · line 1541
Quoted 8 lines
fs/xfs/xfs_buf.c +1
-2@@ -1536,8 +1536,7 @@ __xfs_buf_submit(xfs_buf_ioerror (bp, -EIO ); bp->b_flags &= ~XBF_DONE ;xfs_buf_stale (bp); -if (bp->b_flags &XBF_ASYNC ) -xfs_buf_ioend (bp); +xfs_buf_ioend (bp);return -EIO ; }That said, it definitely looks like it fixes a bug. Will test.
Quoted 2 lines
Reviewed-by Dave Chinner <[email protected]>
Thanks.
Brian
Quoted 4 lines
-Dave.
Signature · 2 lines
Dave Chinner [email protected]