xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror
PATCH 1/1 xfs: Fix re-use of EWOULDBLOCK during read on dm-mirror
When using lvconvert to convert a linear mapping to a dm-raid1 mirror, we encountered issues where the log would be flooded with messages like:
metadata I/O error: block 0xee7060 ("xfs_trans_read_buf") error 11 numblks 8
The cause is that dm-mirror (and striping, and others) will return -EWOULDBLOCK for readahead requests while the mirror is rebuilding. XFS's end_io routine caches the errno and then xfs_buf_iowait bails out early when it encounters it after issuing the i/o request. The I/O eventually succeeds and the endio routine resets bp->b_error, but the original read request has already returned -EWOULDBLOCK to the user and added the log message above to the kernel log, freaking everyone out.
This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O to complete and tries again, allowing the read to succeed as expected.
--- fs/xfs/xfs_buf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/xfs/xfs_buf.c +1 -1
@@ -1252,7 +1252,7 @@ xfs_buf_iowait( {trace_xfs_buf_iowait (bp, _RET_IP_); -if (!bp->b_error) +if (!bp->b_error || bp->b_error ==EWOULDBLOCK )wait_for_completion (&bp->b_iowait);trace_xfs_buf_iowait_done (bp, _RET_IP_);
Signature · 7 lines
Jeff Mahoney SUSE Labs _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 7 lines · Jeff Mahoney
When using lvconvert to convert a linear mapping to a dm-raid1 mirror, we encountered issues where the log would be flooded with messages like:
metadata I/O error: block
0xee7060("xfs_trans_read_buf") error 11 numblks 8The cause is that dm-mirror (and striping, and others) will return
-EWOULDBLOCKfor readahead requests while the mirror is rebuilding.
That's nasty - since when has DM been doing this? I doubt anything handles a EAGAIN error from the storage layer properly - it's not an error the filesystem expects from the lower layers at all.
Quoted 3 lines
XFS's
end_ioroutine caches the errno and thenxfs_buf_iowaitbails out early when it encounters it after issuing the i/o request.
That doesn't sound right. when XFS issues buffer readahead, it does not wait for it to complete. i.e. we never get to xfs_buf_iowait() on readahead buffers.
If something then issues a read on the buffer that failed the readahead, then we enter xfs_buf_iowait() after reissuing the IO. If it's aborting because of a stale EWOULDBLOCK as a result of readahead, then the problem is either:
- failed readahead should not be leaving an error in b_error; or
- the read IO did not zero b_error before starting the IO
Quoted 2 lines
The I/O eventually
succeeds and the endio routine resetsbp->b_error,
AFAICT, it's a different IO that succeeds (i.e. the resubmitted one that is being waited for), not the same one.
Quoted 6 lines
but the original read
request has already returned-EWOULDBLOCKto the user and added the log message above to the kernel log, freaking everyone out.This patch ignores EWOULDBLOCK when deciding whether to wait for the I/O to complete and tries again, allowing the read to succeed as expected.
Which does not appear to be the correct fix - preventing failed readahead from leaving a stale error on the buffer seems like the right thing to do here...
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 4 lines · Dave Chinner
Quoted 3 lines · Jeff Mahoney
When using lvconvert to convert a linear mapping to a dm-raid1 mirror, we encountered issues where the log would be flooded with messages like:
Quoted 8 lines
Quoted 4 lines
metadata I/O error: block
0xee7060("xfs_trans_read_buf") error 11 numblks 8The cause is that dm-mirror (and striping, and others) will return
-EWOULDBLOCKfor readahead requests while the mirror is rebuilding.That's nasty - since when has DM been doing this? I doubt anything handles a EAGAIN error from the storage layer properly - it's not an error the filesystem expects from the lower layers at all.
Wow, it's been doing this since at least 2008 - it's taken the best part of 5 years for someone to trip over this.....
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs