xfs: flush related error handling cleanups
v2 cover xfs: flush related error handling cleanups
Hi all,
Here's a v2 of the various error handling cleanup patches. I opted to retain and slightly rework the xfs_qm_dqflush() error handling patch into something that seems a bit more graceful by simply expanding the error path to include the buffer association. The dqflush verifier check has been fixed up to cover the in-core structure instead of being removed (note that this results in a small tweak to patch 7, but I retained the R-b tags from v1). Finally, a couple new patches are inserted to combine the AIL item removal functions and remove unused shutdown types. Various other small changes are noted in the changelog below.
Thoughts, reviews, flames appreciated.
Brian
git repo: https://github.com/bsfost/linux-xfs/tree/xfs-flush-error-handling-cleanups-v2
v2:
- Rename some helper functions.
- Fix up dquot flush verifier instead of removing it.
- Drop quotaoff push handler removal patch.
- Reuse existing ratelimit state for buffer error messages.
- Combine AIL removal helpers.
- Refactor iflush error handling rework to update log item.
- Remove unused shutdown types.
v1: https://lore.kernel.org/linux-xfs/[email protected]/
Brian Foster (13):
xfs: refactor failed buffer resubmission into xfsaild
xfs: factor out buffer I/O failure simulation code
xfs: fallthru to buffer attach on error and simplify error handling
xfs: remove unnecessary shutdown check from xfs_iflush()
xfs: ratelimit unmount time per-buffer I/O error message
xfs: fix duplicate verification from xfs_qm_dqflush()
xfs: abort consistently on dquot flush failure
xfs: elide the AIL lock on log item failure tracking
xfs: clean up AIL log item removal functions
xfs: combine xfs_trans_ail_[remove|delete]()
xfs: remove unused iflush stale parameter
xfs: random buffer write failure errortag
xfs: remove unused shutdown types
fs/xfs/libxfs/xfs_errortag.h | 4 +- fs/xfs/libxfs/xfs_inode_buf.c | 7 +- fs/xfs/xfs_bmap_item.c | 2 +- fs/xfs/xfs_buf.c | 42 ++++++++++-- fs/xfs/xfs_buf.h | 2 + fs/xfs/xfs_buf_item.c | 96 ++++---------------------- fs/xfs/xfs_buf_item.h | 2 - fs/xfs/xfs_dquot.c | 84 +++++++++-------------- fs/xfs/xfs_dquot_item.c | 17 +---- fs/xfs/xfs_error.c | 3 + fs/xfs/xfs_extfree_item.c | 2 +- fs/xfs/xfs_fsops.c | 5 +- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 124 ++++++++++------------------------ fs/xfs/xfs_inode_item.c | 39 ++--------- fs/xfs/xfs_inode_item.h | 2 +- fs/xfs/xfs_mount.h | 2 - fs/xfs/xfs_refcount_item.c | 2 +- fs/xfs/xfs_rmap_item.c | 2 +- fs/xfs/xfs_trans_ail.c | 79 ++++++++++++++++------ fs/xfs/xfs_trans_priv.h | 23 +------ 21 files changed, 200 insertions(+), 341 deletions(-)
Signature · 1 line
2.21.1
PATCH v2 1/13 xfs: refactor failed buffer resubmission into xfsaild
Flush locked log items whose underlying buffers fail metadata writeback are tagged with a special flag to indicate that the flush lock is already held. This is currently implemented in the type specific ->iop_push() callback, but the processing required for such items is not type specific because we're only doing basic state management on the underlying buffer.
Factor the failed log item handling out of the inode and dquot ->iop_push() callbacks and open code the buffer resubmit helper into a single helper called from xfsaild_push_item(). This provides a generic mechanism for handling failed metadata buffer writeback with a bit less code.
--- fs/xfs/xfs_buf_item.c | 39 --------------------------------------- fs/xfs/xfs_buf_item.h | 2 -- fs/xfs/xfs_dquot_item.c | 15 --------------- fs/xfs/xfs_inode_item.c | 15 --------------- fs/xfs/xfs_trans_ail.c | 41 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 41 insertions(+), 71 deletions(-)
fs/xfs/xfs_buf_item.c +0 -39
@@ -1248,42 +1248,3 @@ xfs_buf_iodone(xfs_trans_ail_delete (ailp, lip,SHUTDOWN_CORRUPT_INCORE );xfs_buf_item_free (BUF_ITEM (lip)); } - -/* -* Requeue a failed buffer for writeback. -* -* We clear the log item failed state here as well, but we have to be careful -* about reference counts because the only active reference counts on the buffer -* may be the failed log items. Hence if we clear the log item failed state -* before queuing the buffer for IO we can release all active references to -* the buffer and free it, leading to use after free problems in -* xfs_buf_delwri_queue. It makes no difference to the buffer or log items which -* order we process them in - the buffer is locked, and we own the buffer list -* so nothing on them is going to change while we are performing this action. -* -* Hence we can safely queue the buffer for IO before we clear the failed log -* item state, therefore always having an active reference to the buffer and -* avoiding the transient zero-reference state that leads to use-after-free. -* -* Return true if the buffer was added to the buffer list, false if it was -* already on the buffer list. -*/ -bool -xfs_buf_resubmit_failed_buffers ( -struct xfs_buf *bp, -struct list_head *buffer_list) -{ -struct xfs_log_item *lip; -bool ret; - - ret =xfs_buf_delwri_queue (bp, buffer_list); - -/* -* XFS_LI_FAILED set/clear is protected by ail_lock, caller of this -* function already have it acquired -*/ -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) -xfs_clear_li_failed (lip); - -return ret; -}
fs/xfs/xfs_buf_item.h +0 -2
@@ -59,8 +59,6 @@ void xfs_buf_attach_iodone(struct xfs_buf *, struct xfs_log_item *);void xfs_buf_iodone_callbacks (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *,struct xfs_log_item *); -bool xfs_buf_resubmit_failed_buffers (struct xfs_buf *, -struct list_head *);bool xfs_buf_log_check_iovec (struct xfs_log_iovec *iovec);extern kmem_zone_t *xfs_buf_item_zone;
fs/xfs/xfs_dquot_item.c +0 -15
@@ -145,21 +145,6 @@ xfs_qm_dquot_logitem_push(if (atomic_read (&dqp->q_pincount) >0 )return XFS_ITEM_PINNED ; -/* -* The buffer containing this item failed to be written back -* previously. Resubmit the buffer for IO -*/ -if (test_bit (XFS_LI_FAILED , &lip->li_flags)) { -if (!xfs_buf_trylock (bp)) -return XFS_ITEM_LOCKED ; - -if (!xfs_buf_resubmit_failed_buffers (bp, buffer_list)) - rval =XFS_ITEM_FLUSHING ; - -xfs_buf_unlock (bp); -return rval; - } -if (!xfs_dqlock_nowait (dqp))return XFS_ITEM_LOCKED ;
fs/xfs/xfs_inode_item.c +0 -15
@@ -497,21 +497,6 @@ xfs_inode_item_push(if (xfs_ipincount (ip) >0 )return XFS_ITEM_PINNED ; -/* -* The buffer containing this item failed to be written back -* previously. Resubmit the buffer for IO. -*/ -if (test_bit (XFS_LI_FAILED , &lip->li_flags)) { -if (!xfs_buf_trylock (bp)) -return XFS_ITEM_LOCKED ; - -if (!xfs_buf_resubmit_failed_buffers (bp, buffer_list)) - rval =XFS_ITEM_FLUSHING ; - -xfs_buf_unlock (bp); -return rval; - } -if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED ))return XFS_ITEM_LOCKED ;
fs/xfs/xfs_trans_ail.c +41 -0
53 lines not on this page · raw message
Signature · 1 line
2.21.1
Re: [PATCH v2 01/13] xfs: refactor failed buffer resubmission into xfsaild
Quoted 7 lines · Brian Foster
Flush locked log items whose underlying buffers fail metadata writeback are tagged with a special flag to indicate that the flush lock is already held. This is currently implemented in the type specific ->iop_push() callback, but the processing required for such items is not type specific because we're only doing basic state management on the underlying buffer.
Quoted 8 lines
Factor the failed log item handling out of the inode and dquot ->iop_push() callbacks and open code the buffer resubmit helper into a single helper called from
xfsaild_push_item(). This provides a generic mechanism for handling failed metadata buffer writeback with a bit less code.Signed-off-by Brian Foster <[email protected]>Reviewed-by Allison Collins <[email protected]>
Looks good.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 01/13] xfs: refactor failed buffer resubmission into xfsaild
In reply to Brian FosterLooks good,
PATCH v2 2/13 xfs: factor out buffer I/O failure simulation code
In reply to Brian FosterWe use the same buffer I/O failure simulation code in a few different places. It's not much code, but it's not necessarily self-explanatory. Factor it into a helper and document it in one place.
--- fs/xfs/xfs_buf.c | 23 +++++++++++++++++++---- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_buf_item.c | 22 +++------------------- fs/xfs/xfs_inode.c | 7 +------ 4 files changed, 24 insertions(+), 29 deletions(-)
fs/xfs/xfs_buf.c +19 -4
@@ -1248,6 +1248,24 @@ xfs_buf_ioerror_alert( -bp->b_error); } +/* +* To simulate an I/O failure, the buffer must be locked and held with at least +* three references. The LRU reference is dropped by the stale call. The buf +* item reference is dropped via ioend processing. The third reference is owned +* by the caller and is dropped on I/O completion if the buffer is XBF_ASYNC. +*/ +void +xfs_buf_ioend_fail ( +struct xfs_buf *bp, +int flags) +{ + bp->b_flags |= flags; + bp->b_flags &= ~XBF_DONE ; +xfs_buf_stale (bp); +xfs_buf_ioerror (bp, -EIO ); +xfs_buf_ioend (bp); +} +int xfs_bwrite (struct xfs_buf *bp)
@@ -1480,10 +1498,7 @@ __xfs_buf_submit(/* on shutdown we stale and complete the buffer immediately */ if (XFS_FORCED_SHUTDOWN (bp->b_mount)) { -xfs_buf_ioerror (bp, -EIO ); - bp->b_flags &= ~XBF_DONE ; -xfs_buf_stale (bp); -xfs_buf_ioend (bp); +xfs_buf_ioend_fail (bp,0 );return -EIO ; }
fs/xfs/xfs_buf.h +1 -0
@@ -263,6 +263,7 @@ extern void __xfs_buf_ioerror(struct xfs_buf *bp, int error, xfs_failaddr_t failaddr);#define xfs_buf_ioerror (bp, err) __xfs_buf_ioerror((bp), (err), __this_address)extern void xfs_buf_ioerror_alert (struct xfs_buf *bp,xfs_failaddr_t fa); +void xfs_buf_ioend_fail (struct xfs_buf *,int );extern int __xfs_buf_submit (struct xfs_buf *bp,bool );static inline int xfs_buf_submit (struct xfs_buf *bp)
fs/xfs/xfs_buf_item.c +3 -19
@@ -471,28 +471,12 @@ xfs_buf_item_unpin(xfs_buf_relse (bp); }else if (freed && remove) {/* -* There are currently two references to the buffer - the active -* LRU reference and the buf log item. What we are about to do -* here - simulate a failed IO completion - requires 3 -* references. -* -* The LRU reference is removed by the xfs_buf_stale() call. The -* buf item reference is removed by the xfs_buf_iodone() -* callback that is run by xfs_buf_do_callbacks() during ioend -* processing (via the bp->b_iodone callback), and then finally -* the ioend processing will drop the IO reference if the buffer -* is marked XBF_ASYNC. -* -* Hence we need to take an additional reference here so that IO -* completion processing doesn't free the buffer prematurely. +* The buffer must be locked and held by the caller to simulate +* an async I/O failure. */ xfs_buf_lock (bp);xfs_buf_hold (bp); - bp->b_flags |=XBF_ASYNC ; -xfs_buf_ioerror (bp, -EIO ); - bp->b_flags &= ~XBF_DONE ; -xfs_buf_stale (bp); -xfs_buf_ioend (bp); +xfs_buf_ioend_fail (bp,XBF_ASYNC ); } }
fs/xfs/xfs_inode.c +1 -6
@@ -3629,12 +3629,7 @@ xfs_iflush_cluster(* xfs_buf_submit(). */ ASSERT (bp->b_iodone); - bp->b_flags |=XBF_ASYNC ; - bp->b_flags &= ~XBF_DONE ; -xfs_buf_stale (bp); -xfs_buf_ioerror (bp, -EIO ); -xfs_buf_ioend (bp); - +xfs_buf_ioend_fail (bp,XBF_ASYNC );xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );/* abort the corrupt inode, as it was not attached to the buffer */
Signature · 1 line
2.21.1
Re: [PATCH v2 02/13] xfs: factor out buffer I/O failure simulation code
Quoted 7 lines · Brian Foster
We use the same buffer I/O failure simulation code in a few different places. It's not much code, but it's not necessarily self-explanatory. Factor it into a helper and document it in one place.
Signed-off-by Brian Foster <[email protected]>Reviewed-by Allison Collins <[email protected]>
Looks good.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 02/13] xfs: factor out buffer I/O failure simulation code
In reply to Brian FosterQuoted 4 lines · Brian Foster
We use the same buffer I/O failure simulation code in a few different places. It's not much code, but it's not necessarily self-explanatory. Factor it into a helper and document it in one place.
The code looks good, but the term simularion sounds rather strange in this context. We don't really simulate an I/O failure, but we fail the buffer with -EIO due to a file system shutdown.
I'd also just keep the ORing of XBF_ASYNC into b_flags in the two callers, as that keeps the function a little more "compact".
Re: [PATCH v2 02/13] xfs: factor out buffer I/O failure simulation code
Quoted 10 lines · Christoph Hellwig
Quoted 4 lines · Brian Foster
We use the same buffer I/O failure simulation code in a few different places. It's not much code, but it's not necessarily self-explanatory. Factor it into a helper and document it in one place.
The code looks good, but the term simularion sounds rather strange in this context. We don't really simulate an I/O failure, but we fail the buffer with
-EIOdue to a file system shutdown.
I was just using the terminology in the existing code. I've updated the commit log to refer to it as "I/O failure code," but note that the comments in the code still use the original terminology (unless you want to suggest alternative wording..).
Quoted 3 lines
I'd also just keep the ORing of
XBF_ASYNCintob_flagsin the two callers, as that keeps the function a little more "compact".
Ok, fine by me.
Brian
PATCH v2 3/13 xfs: fallthru to buffer attach on error and simplify error handling
In reply to Brian FosterThe inode flush code has several layers of error handling between the inode and cluster flushing code. If the inode flush fails before acquiring the backing buffer, the inode flush is aborted. If the cluster flush fails, the current inode flush is aborted and the cluster buffer is failed to handle the initial inode and any others that might have been attached before the error.
Since xfs_iflush() is the only caller of xfs_iflush_cluster(), the error handling between the two can be condensed in the top-level function. If we update xfs_iflush_int() to always fall through to the log item update and attach the item completion handler to the buffer, any errors that occur after the first call to xfs_iflush_int() can be handled with a buffer I/O failure.
Lift the error handling from xfs_iflush_cluster() into xfs_iflush() and consolidate with the existing error handling. This also replaces the need to release the buffer because failing the buffer with XBF_ASYNC drops the current reference.
--- fs/xfs/xfs_inode.c | 108 ++++++++++++++++----------------------------- 1 file changed, 38 insertions(+), 70 deletions(-)
fs/xfs/xfs_inode.c +38 -70
@@ -3496,6 +3496,7 @@ xfs_iflush_cluster(struct xfs_inode **cilist;struct xfs_inode *cip;struct xfs_ino_geometry *igeo =M_IGEO (mp); +int error =0 ;int nr_found;int clcount =0 ;int i;
@@ -3588,11 +3589,10 @@ xfs_iflush_cluster(* re-check that it's dirty before flushing. */ if (!xfs_inode_clean (cip)) { -int error; error =xfs_iflush_int (cip, bp);if (error) {xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto cluster_corrupt_out; + goto out_free; } clcount++; }else {
@@ -3611,32 +3611,7 @@ xfs_iflush_cluster(kmem_free (cilist); out_put:xfs_perag_put (pag); -return 0 ; - - -cluster_corrupt_out: -/* -* Corruption detected in the clustering loop. Invalidate the -* inode buffer and shut down the filesystem. -*/ -rcu_read_unlock (); - -/* -* We'll always have an inode attached to the buffer for completion -* process by the time we are called from xfs_iflush(). Hence we have -* always need to do IO completion processing to abort the inodes -* attached to the buffer. handle them just like the shutdown case in -* xfs_buf_submit(). -*/ -ASSERT (bp->b_iodone); -xfs_buf_ioend_fail (bp,XBF_ASYNC ); -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); - -/* abort the corrupt inode, as it was not attached to the buffer */ -xfs_iflush_abort (cip, false); -kmem_free (cilist); -xfs_perag_put (pag); -return -EFSCORRUPTED ; +return error; }/*
@@ -3692,17 +3667,16 @@ xfs_iflush(*/ if (XFS_FORCED_SHUTDOWN (mp)) { error = -EIO ; - goto abort_out; + goto abort; }/* * Get the buffer containing the on-disk inode. We are doing a try-lock -* operation here, so we may get an EAGAIN error. In that case, we -* simply want to return with the inode still dirty. +* operation here, so we may get an EAGAIN error. In that case, return +* leaving the inode dirty. * * If we get any other error, we effectively have a corruption situation -* and we cannot flush the inode, so we treat it the same as failing -* xfs_iflush_int(). +* and we cannot flush the inode. Abort the flush and shut down. */ error =xfs_imap_to_bp (mp,NULL , &ip->i_imap, &dip, &bp,XBF_TRYLOCK ,0 );
@@ -3711,14 +3685,7 @@ xfs_iflush(return error; }if (error) - goto corrupt_out; - -/* -* First flush out the inode that xfs_iflush was called with. -*/ - error =xfs_iflush_int (ip, bp); -if (error) - goto corrupt_out; + goto abort;/* * If the buffer is pinned then push on the log now so we won't
@@ -3728,28 +3695,28 @@ xfs_iflush(xfs_log_force (mp,0 );/* -* inode clustering: try to gather other inodes into this write +* Flush the provided inode then attempt to gather others from the +* cluster into the write. * -* Note: Any error during clustering will result in the filesystem -* being shut down and completion callbacks run on the cluster buffer. -* As we have already flushed and attached this inode to the buffer, -* it has already been aborted and released by xfs_iflush_cluster() and -* so we have no further error handling to do here. +* Note: Once we attempt to flush an inode, we must run buffer +* completion callbacks on any failure. If this fails, simulate an I/O +* failure on the buffer and shut down. */ - error =xfs_iflush_cluster (ip, bp); -if (error) -return error; + error =xfs_iflush_int (ip, bp); +if (!error) + error =xfs_iflush_cluster (ip, bp); +if (error) { +xfs_buf_ioend_fail (bp,XBF_ASYNC ); + goto shutdown; + } *bpp = bp;return 0 ; -corrupt_out: -if (bp) -xfs_buf_relse (bp); -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); -abort_out: -/* abort the corrupt inode, as it was not attached to the buffer */ +abort:xfs_iflush_abort (ip, false); +shutdown: +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );return error; }
@@ -3791,6 +3758,7 @@ xfs_iflush_int(struct xfs_inode_log_item *iip = ip->i_itemp;struct xfs_dinode *dip;struct xfs_mount *mp = ip->i_mount; +int error;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL |XFS_ILOCK_SHARED ));ASSERT (xfs_isiflocked (ip));
@@ -3801,12 +3769,13 @@ xfs_iflush_int(/* set *dip = inode's place in the buffer */ dip =xfs_buf_offset (bp, ip->i_imap.im_boffset); + error = -EFSCORRUPTED ;if (XFS_TEST_ERROR (dip->di_magic !=cpu_to_be16 (XFS_DINODE_MAGIC ), mp,XFS_ERRTAG_IFLUSH_1 )) {xfs_alert_tag (mp,XFS_PTAG_IFLUSH ,"%s: Bad inode %Lu magic number 0x%x, ptr " PTR_FMT , __func__, ip->i_ino,be16_to_cpu (dip->di_magic), dip); - goto corrupt_out; + goto flush_out; }if (S_ISREG (VFS_I (ip)->i_mode)) { if (XFS_TEST_ERROR(
@@ -3816,7 +3785,7 @@ xfs_iflush_int(xfs_alert_tag (mp,XFS_PTAG_IFLUSH ,"%s: Bad regular inode %Lu, ptr " PTR_FMT , __func__, ip->i_ino, ip); - goto corrupt_out; + goto flush_out; } }else if (S_ISDIR (VFS_I (ip)->i_mode)) { if (XFS_TEST_ERROR(
@@ -3827,7 +3796,7 @@ xfs_iflush_int(xfs_alert_tag (mp,XFS_PTAG_IFLUSH ,"%s: Bad directory inode %Lu, ptr " PTR_FMT , __func__, ip->i_ino, ip); - goto corrupt_out; + goto flush_out; } } if (XFS_TEST_ERROR(ip->i_d.di_nextents + ip->i_d.di_anextents >
@@ -3838,14 +3807,14 @@ xfs_iflush_int( __func__, ip->i_ino, ip->i_d.di_nextents + ip->i_d.di_anextents, ip->i_d.di_nblocks, ip); - goto corrupt_out; + goto flush_out; }if (XFS_TEST_ERROR (ip->i_d.di_forkoff > mp->m_sb.sb_inodesize, mp,XFS_ERRTAG_IFLUSH_6 )) {xfs_alert_tag (mp,XFS_PTAG_IFLUSH ,"%s: bad inode %Lu, forkoff 0x%x, ptr " PTR_FMT , __func__, ip->i_ino, ip->i_d.di_forkoff, ip); - goto corrupt_out; + goto flush_out; }/*
@@ -3862,7 +3831,7 @@ xfs_iflush_int(/* Check the inline fork data before we write out. */ if (!xfs_inode_verify_forks (ip)) - goto corrupt_out; + goto flush_out;/* * Copy the dirty parts of the inode into the on-disk inode. We always
@@ -3905,6 +3874,8 @@ xfs_iflush_int(* need the AIL lock, because it is a 64 bit value that cannot be read * atomically. */ + error =0 ; +flush_out: iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ;
@@ -3914,10 +3885,10 @@ xfs_iflush_int( &iip->ili_item.li_lsn);/* -* Attach the function xfs_iflush_done to the inode's -* buffer. This will remove the inode from the AIL -* and unlock the inode's flush lock when the inode is -* completely written to disk. +* Attach the inode item callback to the buffer whether the flush +* succeeded or not. If not, the caller will shut down and fail I/O +* completion on the buffer to remove the inode from the AIL and release +* the flush lock. */ xfs_buf_attach_iodone (bp, xfs_iflush_done, &iip->ili_item);
@@ -3926,10 +3897,7 @@ xfs_iflush_int(ASSERT (!list_empty (&bp->b_li_list));ASSERT (bp->b_iodone !=NULL ); -return 0 ; - -corrupt_out: -return -EFSCORRUPTED ; +return error; }/* Release an inode. */
Signature · 1 line
2.21.1
Re: [PATCH v2 03/13] xfs: fallthru to buffer attach on error and simplify error handling
Quoted 14 lines · Brian Foster
The inode flush code has several layers of error handling between the inode and cluster flushing code. If the inode flush fails before acquiring the backing buffer, the inode flush is aborted. If the cluster flush fails, the current inode flush is aborted and the cluster buffer is failed to handle the initial inode and any others that might have been attached before the error.
Since
xfs_iflush()is the only caller ofxfs_iflush_cluster(), the error handling between the two can be condensed in the top-level function. If we updatexfs_iflush_int()to always fall through to the log item update and attach the item completion handler to the buffer, any errors that occur after the first call toxfs_iflush_int()can be handled with a buffer I/O failure.
Quoted 6 lines
Lift the error handling from
xfs_iflush_cluster()intoxfs_iflush()and consolidate with the existing error handling. This also replaces the need to release the buffer because failing the buffer withXBF_ASYNCdrops the current reference.Signed-off-by Brian Foster <[email protected]>
Needs a better subject line, because I had no idea what it meant until I got to the last hunks in the patch. Perhaps: "Simplify inode flush error handling" would be a better summary of the patch....
fs/xfs/xfs_inode.c xfs_iflush_int( · line 3761
Quoted 5 lines
fs/xfs/xfs_inode.c +1
-0@@ -3791,6 +3758,7 @@ xfs_iflush_int(struct xfs_inode_log_item *iip = ip->i_itemp;struct xfs_dinode *dip;struct xfs_mount *mp = ip->i_mount; +int error;
There needs to be a comment added to this function to explain why we always attached the inode to the buffer and update the flush state, even on error. This:
Quoted 4 lines
fs/xfs/xfs_inode.c +0
-0@@ -3914,10 +3885,10 @@ xfs_iflush_int( &iip->ili_item.li_lsn);/*
fs/xfs/xfs_inode.c xfs_iflush_int( · line 3893
Quoted 10 lines
fs/xfs/xfs_inode.c +4
-4-* Attach the function xfs_iflush_done to the inode's -* buffer. This will remove the inode from the AIL -* and unlock the inode's flush lock when the inode is -* completely written to disk. +* Attach the inode item callback to the buffer whether the flush +* succeeded or not. If not, the caller will shut down and fail I/O +* completion on the buffer to remove the inode from the AIL and release +* the flush lock. */ xfs_buf_attach_iodone (bp, xfs_iflush_done, &iip->ili_item);
isn't obviously associated with the "flush_out" label, and so the structure of the function really isn't explained until you get to the end of the function. And that's still easy to miss...
Other than that, the code looks OK.
CHeers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 03/13] xfs: fallthru to buffer attach on error and simplify error handling
Quoted 20 lines · Dave Chinner
Quoted 19 lines · Brian Foster
The inode flush code has several layers of error handling between the inode and cluster flushing code. If the inode flush fails before acquiring the backing buffer, the inode flush is aborted. If the cluster flush fails, the current inode flush is aborted and the cluster buffer is failed to handle the initial inode and any others that might have been attached before the error.
Since
xfs_iflush()is the only caller ofxfs_iflush_cluster(), the error handling between the two can be condensed in the top-level function. If we updatexfs_iflush_int()to always fall through to the log item update and attach the item completion handler to the buffer, any errors that occur after the first call toxfs_iflush_int()can be handled with a buffer I/O failure.Lift the error handling from
xfs_iflush_cluster()intoxfs_iflush()and consolidate with the existing error handling. This also replaces the need to release the buffer because failing the buffer withXBF_ASYNCdrops the current reference.
Quoted 7 lines
Quoted 1 line
Signed-off-by Brian Foster <[email protected]>Needs a better subject line, because I had no idea what it meant until I got to the last hunks in the patch. Perhaps: "Simplify inode flush error handling" would be a better summary of the patch....
Ok, fixed.
Quoted 10 lines
Review xfs_iflush_int( · line 3761
Quoted 5 lines
(quoted diff) +1
-0@@ -3791,6 +3758,7 @@ xfs_iflush_int( struct xfs_inode_log_item *iip = ip->i_itemp; struct xfs_dinode *dip; struct xfs_mount *mp = ip->i_mount; + int error;There needs to be a comment added to this function to explain why we always attached the inode to the buffer and update the flush state, even on error. This:
Indeed. Updated as follows with a comment before the first corruption check:
fs/xfs/xfs_inode.c +6 -1
@@ -3766,9 +3766,14 @@ xfs_iflush_int( ip->i_d.di_nextents > XFS_IFORK_MAXEXT(ip, XFS_DATA_FORK));ASSERT (iip !=NULL && iip->ili_fields !=0 ); -/* set *dip = inode's place in the buffer */ dip =xfs_buf_offset (bp, ip->i_imap.im_boffset); +/* +* We don't flush the inode if any of the following checks fail, but we +* do still update the log item and attach to the backing buffer as if +* the flush happened. This is a formality to facilitate predictable +* error handling as the caller will shutdown and fail the buffer. +*/ error = -EFSCORRUPTED ;if (XFS_TEST_ERROR (dip->di_magic !=cpu_to_be16 (XFS_DINODE_MAGIC ), mp,XFS_ERRTAG_IFLUSH_1 )) {
Brian
Quoted 28 lines
Quoted 4 lines
(quoted diff) +0
-0@@ -3914,10 +3885,10 @@ xfs_iflush_int( &iip->ili_item.li_lsn); /*Review xfs_iflush_int( · line 3893
Quoted 10 lines
(quoted diff) +4
-4- * Attach the function xfs_iflush_done to the inode's - * buffer. This will remove the inode from the AIL - * and unlock the inode's flush lock when the inode is - * completely written to disk. + * Attach the inode item callback to the buffer whether the flush + * succeeded or not. If not, the caller will shut down and fail I/O + * completion on the buffer to remove the inode from the AIL and release + * the flush lock. */ xfs_buf_attach_iodone(bp, xfs_iflush_done, &iip->ili_item);isn't obviously associated with the "
flush_out" label, and so the structure of the function really isn't explained until you get to the end of the function. And that's still easy to miss...Other than that, the code looks OK.
CHeers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 03/13] xfs: fallthru to buffer attach on error and simplify error handling
In reply to Brian FosterI agree with Dave that the subject sounds rather odd. The actual change looks good to me:
PATCH v2 4/13 xfs: remove unnecessary shutdown check from xfs_iflush()
In reply to Brian FosterThe shutdown check in xfs_iflush() duplicates checks down in the buffer code. If the fs is shut down, xfs_trans_read_buf_map() always returns an error and falls into the same error path. Remove the unnecessary check along with the warning in xfs_imap_to_bp() that generates excessive noise in the log if the fs is shut down.
--- fs/xfs/libxfs/xfs_inode_buf.c | 7 +------ fs/xfs/xfs_inode.c | 13 ------------- 2 files changed, 1 insertion(+), 19 deletions(-)
fs/xfs/libxfs/xfs_inode_buf.c +1 -6
@@ -172,12 +172,7 @@ xfs_imap_to_bp( (int )imap->im_len, buf_flags, &bp, &xfs_inode_buf_ops);if (error) { -if (error == -EAGAIN ) { -ASSERT (buf_flags &XBF_TRYLOCK ); -return error; - } -xfs_warn (mp,"%s: xfs_trans_read_buf() returned error %d." , - __func__, error); +ASSERT (error != -EAGAIN || (buf_flags &XBF_TRYLOCK ));return error; }
fs/xfs/xfs_inode.c +0 -13
@@ -3657,19 +3657,6 @@ xfs_iflush(return 0 ; } -/* -* This may have been unpinned because the filesystem is shutting -* down forcibly. If that's the case we must not write this inode -* to disk, because the log record didn't make it to disk. -* -* We also have to remove the log item from the AIL in this case, -* as we wait for an empty AIL as part of the unmount process. -*/ -if (XFS_FORCED_SHUTDOWN (mp)) { - error = -EIO ; - goto abort; - } -/* * Get the buffer containing the on-disk inode. We are doing a try-lock * operation here, so we may get an EAGAIN error. In that case, return
Signature · 1 line
2.21.1
Re: [PATCH v2 04/13] xfs: remove unnecessary shutdown check from xfs_iflush()
Quoted 9 lines · Brian Foster
The shutdown check in
xfs_iflush()duplicates checks down in the buffer code. If the fs is shut down,xfs_trans_read_buf_map()always returns an error and falls into the same error path. Remove the unnecessary check along with the warning inxfs_imap_to_bp()that generates excessive noise in the log if the fs is shut down.Signed-off-by Brian Foster <[email protected]>Reviewed-by Allison Collins <[email protected]>Reviewed-by Dave Chinner <[email protected]>
Looks good,
PATCH v2 5/13 xfs: ratelimit unmount time per-buffer I/O error message
In reply to Brian FosterAt unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
--- fs/xfs/xfs_buf.c | 13 +++++++++++-- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_buf_item.c | 10 +--------- 3 files changed, 13 insertions(+), 11 deletions(-)
fs/xfs/xfs_buf.c +11 -2
@@ -1684,10 +1684,12 @@ xfs_wait_buftarg(struct xfs_buf *bp; bp = list_first_entry(&dispose, struct xfs_buf, b_lru);list_del_init (&bp->b_lru); -if (bp->b_flags &XBF_WRITE_FAIL ) { +if (bp->b_flags &XBF_WRITE_FAIL && +___ratelimit (&bp->b_target->bt_ioerror_rl, +"XFS: Corruption Alert" )) {xfs_alert (btp->bt_mount,"Corruption Alert: Buffer at daddr 0x%llx had permanent write failures!" , - (long long )bp->b_bn); + (long long )bp->b_bn);xfs_alert (btp->bt_mount,"Please run xfs_repair to determine the extent of the problem." ); }
@@ -1828,6 +1830,13 @@ xfs_alloc_buftarg( btp->bt_bdev = bdev; btp->bt_daxdev = dax_dev; +/* +* Buffer IO error rate limiting. Limit it to no more than 10 messages +* per 30 seconds so as to not spam logs too much on repeated errors. +*/ +ratelimit_state_init (&btp->bt_ioerror_rl,30 *HZ , +DEFAULT_RATELIMIT_BURST ); +if (xfs_setsize_buftarg_early (btp, bdev)) goto error_free;
fs/xfs/xfs_buf.h +1 -0
@@ -91,6 +91,7 @@ typedef struct xfs_buftarg {struct list_lru bt_lru;struct percpu_counter bt_io_count; +struct ratelimit_state bt_ioerror_rl; } xfs_buftarg_t;struct xfs_buf ;
fs/xfs/xfs_buf_item.c +1 -9
@@ -480,14 +480,6 @@ xfs_buf_item_unpin( } } -/* -* Buffer IO error rate limiting. Limit it to no more than 10 messages per 30 -* seconds so as to not spam logs too much on repeated detection of the same -* buffer being bad.. -*/ - -static DEFINE_RATELIMIT_STATE(xfs_buf_write_fail_rl_state, 30 * HZ, 10); - STATIC uint xfs_buf_item_push( struct xfs_log_item *lip,
@@ -518,7 +510,7 @@ xfs_buf_item_push(/* has a previous flush failed due to IO errors? */ if ((bp->b_flags &XBF_WRITE_FAIL ) && -___ratelimit (&xfs_buf_write_fail_rl_state,"XFS: Failing async write" )) { +___ratelimit (&bp->b_target->bt_ioerror_rl,"XFS: Failing async write" )) {xfs_warn (bp->b_mount,"Failing async write on buffer block 0x%llx. Retrying async write." , (long long )bp->b_bn);
Signature · 1 line
2.21.1
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>
Looks fine, but I suspect we both missed something here:xfs_buf_ioerror_alert() was made a ratelimited printk in the last cycle:
void
xfs_buf_ioerror_alert(
struct xfs_buf *bp,
xfs_failaddr_t func)
{
xfs_alert_ratelimited(bp->b_mount,
"metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d",
func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length,
-bp->b_error);
}Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in xfs_message.c:
void
xfs_buf_alert_ratelimited(
struct xfs_buf *bp,
const char *rlmsg,
const char *fmt,
...)
{
struct va_format vaf;
va_list args;
if (!___ratelimit(&bp->b_target->bt_ioerror_rl, rlmsg)
return;
va_start(args, fmt);
vaf.fmt = fmt;
vaf.args = &args;
__xfs_printk(KERN_ALERT, bp->b_mount, &vaf);
va_end(args);
}and:
void
xfs_buf_ioerror_alert(
struct xfs_buf *bp,
xfs_failaddr_t func)
{
xfs_buf_alert_ratelimited(bp, "XFS: metadata IO error",
"metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d",
func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error);
}Quoted 15 lines
--- fs/xfs/xfs_buf.c | 13 +++++++++++-- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_buf_item.c | 10 +--------- 3 files changed, 13 insertions(+), 11 deletions(-)fs/xfs/xfs_buf.c +0
-1@@ -1684,10 +1684,12 @@ xfs_wait_buftarg(struct xfs_buf *bp; bp = list_first_entry(&dispose, struct xfs_buf, b_lru);list_del_init (&bp->b_lru); -if (bp->b_flags &XBF_WRITE_FAIL ) {
fs/xfs/xfs_buf.c xfs_wait_buftarg( · line 1695
Quoted 10 lines
fs/xfs/xfs_buf.c +4
-1+if (bp->b_flags &XBF_WRITE_FAIL && +___ratelimit (&bp->b_target->bt_ioerror_rl, +"XFS: Corruption Alert" )) {xfs_alert (btp->bt_mount,"Corruption Alert: Buffer at daddr 0x%llx had permanent write failures!" , - (long long )bp->b_bn); + (long long )bp->b_bn);xfs_alert (btp->bt_mount,"Please run xfs_repair to determine the extent of the problem." ); }
I think if we are tossing away metadata here, we should probably shut down the filesystem once the loop has completed. That way we get all the normal warnings about running xfs_repair and don't have to open code it here...
fs/xfs/xfs_buf.c xfs_buf_item_push( · line 516
Quoted 13 lines
fs/xfs/xfs_buf.c +1
-2- STATIC uint xfs_buf_item_push( struct xfs_log_item *lip,@@ -518,7 +510,7 @@ xfs_buf_item_push(/* has a previous flush failed due to IO errors? */ if ((bp->b_flags &XBF_WRITE_FAIL ) && -___ratelimit (&xfs_buf_write_fail_rl_state,"XFS: Failing async write" )) { +___ratelimit (&bp->b_target->bt_ioerror_rl,"XFS: Failing async write" )) {xfs_warn (bp->b_mount,"Failing async write on buffer block 0x%llx. Retrying async write." , (long long )bp->b_bn);
This gets simplified to:
if (bp->b_flags & XBF_WRITE_FAIL) {
xfs_buf_alert_ratelimited(bp, "XFS: Failing async write",
"Failing async write on buffer block 0x%llx. Retrying async write.",
(long long)bp->b_bn);
}Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 22 lines · Dave Chinner
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>Looks fine, but I suspect we both missed something here:
xfs_buf_ioerror_alert()was made a ratelimited printk in the last cycle:void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc)
Quoted 7 lines
{
xfs_alert_ratelimited(bp->b_mount,
"metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d",
func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length,
-bp->b_error);
}Yeah, I hadn't noticed that.
Quoted 3 lines
Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in
xfs_message.c:
One thing to note is that xfs_alert_ratelimited() ultimately uses the DEFAULT_RATELIMIT_INTERVAL of 5s. The ratelimit we're generalizing here uses 30s (both use a burst of 10). That seems reasonable enough to me for I/O errors so I'm good with the changes below.
FWIW, that also means we could just call xfs_buf_alert_ratelimited() from xfs_buf_item_push() if we're also Ok with using an "alert" instead of a "warn." I'm not immediately aware of a reason to use one over the other (xfs_wait_buftarg() already uses alert) so I'll try that unless I hear an objection. The xfs_wait_buftarg() ratelimit presumably remains open coded because it's two separate calls and we probably don't want them to individually count against the limit.
Brian
Quoted 93 lines
void
xfs_buf_alert_ratelimited(
structxfs_buf*bp,
const char *rlmsg,
const char *fmt,
...){ struct va_format vaf; va_list args; if (!___ratelimit(&bp->b_target->bt_ioerror_rl, rlmsg) return; va_start(args, fmt); vaf.fmt = fmt; vaf.args = &args; __xfs_printk(KERN_ALERT, bp->b_mount, &vaf); va_end(args); }and:
void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc){ xfs_buf_alert_ratelimited(bp, "XFS: metadata IO error", "metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d", func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error); }Quoted 15 lines
--- fs/xfs/xfs_buf.c | 13 +++++++++++-- fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_buf_item.c | 10 +--------- 3 files changed, 13 insertions(+), 11 deletions(-)fs/xfs/xfs_buf.c +0
-1@@ -1684,10 +1684,12 @@ xfs_wait_buftarg(struct xfs_buf *bp; bp = list_first_entry(&dispose, struct xfs_buf, b_lru);list_del_init (&bp->b_lru); -if (bp->b_flags &XBF_WRITE_FAIL ) {fs/xfs/xfs_buf.c xfs_wait_buftarg( · line 1695
Quoted 10 lines
fs/xfs/xfs_buf.c +4
-1+if (bp->b_flags &XBF_WRITE_FAIL && +___ratelimit (&bp->b_target->bt_ioerror_rl, +"XFS: Corruption Alert" )) {xfs_alert (btp->bt_mount,"Corruption Alert: Buffer at daddr 0x%llx had permanent write failures!" , - (long long )bp->b_bn); + (long long )bp->b_bn);xfs_alert (btp->bt_mount,"Please run xfs_repair to determine the extent of the problem." ); }I think if we are tossing away metadata here, we should probably shut down the filesystem once the loop has completed. That way we get all the normal warnings about running
xfs_repairand don't have to open code it here...fs/xfs/xfs_buf.c xfs_buf_item_push( · line 516
Quoted 13 lines
fs/xfs/xfs_buf.c +1
-2- STATIC uint xfs_buf_item_push( struct xfs_log_item *lip,@@ -518,7 +510,7 @@ xfs_buf_item_push(/* has a previous flush failed due to IO errors? */ if ((bp->b_flags &XBF_WRITE_FAIL ) && -___ratelimit (&xfs_buf_write_fail_rl_state,"XFS: Failing async write" )) { +___ratelimit (&bp->b_target->bt_ioerror_rl,"XFS: Failing async write" )) {xfs_warn (bp->b_mount,"Failing async write on buffer block 0x%llx. Retrying async write." , (long long )bp->b_bn);This gets simplified to:
if (bp->b_flags & XBF_WRITE_FAIL) { xfs_buf_alert_ratelimited(bp, "XFS: Failing async write", "Failing async write on buffer block 0x%llx. Retrying async write.", (long long)bp->b_bn); }Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 42 lines · Brian Foster
Quoted 22 lines · Dave Chinner
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>Looks fine, but I suspect we both missed something here:
xfs_buf_ioerror_alert()was made a ratelimited printk in the last cycle:void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc)Quoted 7 lines
{ xfs_alert_ratelimited(bp->b_mount, "metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d", func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error); }Yeah, I hadn't noticed that.
Quoted 3 lines
Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in
xfs_message.c:One thing to note is that
xfs_alert_ratelimited()ultimately uses theDEFAULT_RATELIMIT_INTERVALof 5s. The ratelimit we're generalizing here uses 30s (both use a burst of 10). That seems reasonable enough to me for I/O errors so I'm good with the changes below.
Quoted 5 lines
FWIW, that also means we could just call
xfs_buf_alert_ratelimited()fromxfs_buf_item_push()if we're also Ok with using an "alert" instead of a "warn." I'm not immediately aware of a reason to use one over the other (xfs_wait_buftarg()already uses alert) so I'll try that unless I hear an objection.
SOunds fine to me.
Quoted 3 lines
The
xfs_wait_buftarg()ratelimit presumably remains
open coded because it's two separate calls and we probably don't want them to individually count against the limit.
That's why I suggested dropping the second "run xfs_repair" message and triggering a shutdown after the wait loop. That way we don't issue "run xfs_repair" for every single failed buffer (largely noise!), and we get a non-rate-limited common "run xfs-repair" message once we processed all the failed writes.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 55 lines · Dave Chinner
Quoted 42 lines · Brian Foster
Quoted 22 lines · Dave Chinner
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>Looks fine, but I suspect we both missed something here:
xfs_buf_ioerror_alert()was made a ratelimited printk in the last cycle:void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc)Quoted 7 lines
{ xfs_alert_ratelimited(bp->b_mount, "metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d", func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error); }Yeah, I hadn't noticed that.
Quoted 3 lines
Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in
xfs_message.c:One thing to note is that
xfs_alert_ratelimited()ultimately uses theDEFAULT_RATELIMIT_INTERVALof 5s. The ratelimit we're generalizing here uses 30s (both use a burst of 10). That seems reasonable enough to me for I/O errors so I'm good with the changes below.Quoted 5 lines
FWIW, that also means we could just call
xfs_buf_alert_ratelimited()fromxfs_buf_item_push()if we're also Ok with using an "alert" instead of a "warn." I'm not immediately aware of a reason to use one over the other (xfs_wait_buftarg()already uses alert) so I'll try that unless I hear an objection.SOunds fine to me.
Quoted 3 lines
The
xfs_wait_buftarg()ratelimit presumably remains
open coded because it's two separate calls and we probably don't want them to individually count against the limit.
Quoted 6 lines
That's why I suggested dropping the second "run
xfs_repair" message and triggering a shutdown after the wait loop. That way we don't issue "runxfs_repair" for every single failed buffer (largely noise!), and we get a non-rate-limited common "run xfs-repair" message once we processed all the failed writes.
Sorry, must have missed that in the last reply. I don't think we want to shut down here because XBF_WRITE_FAIL only reflects that the internal async write retry (e.g. the one historically used to mitigate transient I/O errors) has occurred on the buffer, not necessarily that the immediately previous I/O has failed. For that reason I've kind of looked at this particular instance as more of a warning that I/O errors have occurred in the past and the user might want to verify it didn't result in unexpected damage. FWIW, I've observed plenty of these messages long after I've disabled error injection and allowed I/O to proceed and the fs to unmount cleanly.
Looking at it again, the wording of the message is much stronger than a warning (i.e. "Corruption Alert", "permanent write failures"), so perhaps we should revisit what we're actually trying to accomplish with this message. Note that we do have the buffer push time alert to indicate that I/O errors and retries are occurring, as well as error handling logic to shutdown on I/O failure while the fs is unmounting (xfs_buf_iodone_callback_error()), so both of those cases are fundamentally covered already.
ISTM that leaves at least a few simple options for the xfs_wait_buftarg() message:
1.) Remove it.
2.) Massage it into more of a "Warning: buffer I/O errors have occurred in the past" type of message. This could perhaps retain the existing XBF_WRITE_FAIL logic, but use a flag to lift it out of the loop and thus avoid the need to rate limit it at all.
3.) Fix up the logic to only dump (ratelimited) specifics on buffers that have actually failed I/O. This means we use the existing message but perhaps check ->b_error instead of XBF_WRITE_FAIL. We still don't need to shut down (I'd probably add an assert), but we could also lift the second xfs_repair line of the message out of the loop to reduce the noise.
We could also look into clearing XBF_WRITE_RETRY on successful I/O completion, FWIW. The existing logic kind of bugs me regardless. Thoughts?
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 63 lines · Brian Foster
Quoted 55 lines · Dave Chinner
Quoted 42 lines · Brian Foster
Quoted 22 lines · Dave Chinner
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>Looks fine, but I suspect we both missed something here:
xfs_buf_ioerror_alert()was made a ratelimited printk in the last cycle:void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc)Quoted 7 lines
{ xfs_alert_ratelimited(bp->b_mount, "metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d", func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error); }Yeah, I hadn't noticed that.
Quoted 3 lines
Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in
xfs_message.c:One thing to note is that
xfs_alert_ratelimited()ultimately uses theDEFAULT_RATELIMIT_INTERVALof 5s. The ratelimit we're generalizing here uses 30s (both use a burst of 10). That seems reasonable enough to me for I/O errors so I'm good with the changes below.Quoted 5 lines
FWIW, that also means we could just call
xfs_buf_alert_ratelimited()fromxfs_buf_item_push()if we're also Ok with using an "alert" instead of a "warn." I'm not immediately aware of a reason to use one over the other (xfs_wait_buftarg()already uses alert) so I'll try that unless I hear an objection.SOunds fine to me.
Quoted 3 lines
The
xfs_wait_buftarg()ratelimit presumably remains
open coded because it's two separate calls and we probably don't want them to individually count against the limit.Quoted 6 lines
That's why I suggested dropping the second "run
xfs_repair" message and triggering a shutdown after the wait loop. That way we don't issue "runxfs_repair" for every single failed buffer (largely noise!), and we get a non-rate-limited common "run xfs-repair" message once we processed all the failed writes.
Quoted 5 lines
Sorry, must have missed that in the last reply. I don't think we want to shut down here because
XBF_WRITE_FAILonly reflects that the internal async write retry (e.g. the one historically used to mitigate transient I/O errors) has occurred on the buffer, not necessarily that the immediately previous I/O has failed.
I think this is an incorrect reading of how XBF_WRITE_FAIL functions. XBF_WRITE_FAIL is used to indicate that the previous write failed, not that a historic write failed. The flag is cleared at buffer submission time - see xfs_buf_delwri_submit_buffers() and xfs_bwrite() - and so it is only set on buffers whose previous IO failed and hence is still dirty and has not been flushed back to disk.
If we hit this in xfs_buftarg_wait() after we've pushed the AIL in xfs_log_quiesce() on unmount, then we've got write failures that could not be resolved by repeated retries, and the filesystem is, at this instant in time, inconsistent on disk.
That's a shutdown error condition...
Quoted 6 lines
For that reason I've kind of looked
at this particular instance as more of a warning that I/O errors have occurred in the past and the user might want to verify it didn't result in unexpected damage. FWIW, I've observed plenty of these messages long after I've disabled error injection and allowed I/O to proceed and the fs to unmount cleanly.
Right. THat's the whole point of the flag - the buffer has been dirtied and it hasn't been able to be written back when we are purging the buffer cache at the end of unmount. i.e. When xfs_buftarg_wait() is called, all buffers should be clean because we are about to write an unmount record to mark the log clean once all the logged metadata is written back.
What we do right now - write an unmount record after failing metadta writeback - is actually a bug, and that is one of the reasons why I suggested a shutdown should be done. i.e. we should not be writing an unmount record to mark the log clean if we failed to write back metadata. That metadata is still valid in the journal, and so it should remain valid in the journal to allow it to be replayed on the next mount. i.e. retry the writes from log recovery after the hardware failure has been addressed and the IO errors have gone away.
Tossing the dirty buffers unmount and then marking the journal clean is actually making the buffer write failure -worse-. Doing this guarantees the filesystem is inconsistent on disk (by updating the journal to indicate those writes actually succeeded) and absolutely requires xfs_repair to fix as a result.
If we shut down on XBF_WRITE_FAIL buffers in xfs_buftarg_wait(), we will not write an unmount record and so give the filesystem a chance to recover on next mount (e.g. after a power cycle to clear whatever raid hardware bug was being hit) and write that dirty metadata back without failure. If recovery fails with IO errors, then the user really does need to run repair. However, the situation at this point is still better than if we write a clean unmount record after write failures...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 05/13] xfs: ratelimit unmount time per-buffer I/O error message
Quoted 70 lines · Dave Chinner
Quoted 63 lines · Brian Foster
Quoted 55 lines · Dave Chinner
Quoted 42 lines · Brian Foster
Quoted 22 lines · Dave Chinner
Quoted 7 lines · Brian Foster
At unmount time, XFS emits a warning for every in-core buffer that might have undergone a write error. In practice this behavior is probably reasonable given that the filesystem is likely short lived once I/O errors begin to occur consistently. Under certain test or otherwise expected error conditions, this can spam the logs and slow down the unmount.
Quoted 5 lines
We already have a ratelimit state defined for buffers failing writeback. Fold this state into the buftarg and reuse it for the unmount time errors.
Signed-off-by Brian Foster <[email protected]>Looks fine, but I suspect we both missed something here:
xfs_buf_ioerror_alert()was made a ratelimited printk in the last cycle:void
xfs_buf_ioerror_alert(
structxfs_buf*bp,
xfs_failaddr_tfunc)Quoted 7 lines
{ xfs_alert_ratelimited(bp->b_mount, "metadata I/O error in \"%pS\" at daddr 0x%llx len %d error %d", func, (uint64_t)XFS_BUF_ADDR(bp), bp->b_length, -bp->b_error); }Yeah, I hadn't noticed that.
Quoted 3 lines
Hence I think all these buffer error alerts can be brought under the same rate limiting variable. Something like this in
xfs_message.c:One thing to note is that
xfs_alert_ratelimited()ultimately uses theDEFAULT_RATELIMIT_INTERVALof 5s. The ratelimit we're generalizing here uses 30s (both use a burst of 10). That seems reasonable enough to me for I/O errors so I'm good with the changes below.Quoted 5 lines
FWIW, that also means we could just call
xfs_buf_alert_ratelimited()fromxfs_buf_item_push()if we're also Ok with using an "alert" instead of a "warn." I'm not immediately aware of a reason to use one over the other (xfs_wait_buftarg()already uses alert) so I'll try that unless I hear an objection.SOunds fine to me.
Quoted 3 lines
The
xfs_wait_buftarg()ratelimit presumably remains
open coded because it's two separate calls and we probably don't want them to individually count against the limit.Quoted 6 lines
That's why I suggested dropping the second "run
xfs_repair" message and triggering a shutdown after the wait loop. That way we don't issue "runxfs_repair" for every single failed buffer (largely noise!), and we get a non-rate-limited common "run xfs-repair" message once we processed all the failed writes.Quoted 5 lines
Sorry, must have missed that in the last reply. I don't think we want to shut down here because
XBF_WRITE_FAILonly reflects that the internal async write retry (e.g. the one historically used to mitigate transient I/O errors) has occurred on the buffer, not necessarily that the immediately previous I/O has failed.
Quoted 8 lines
I think this is an incorrect reading of how
XBF_WRITE_FAILfunctions.XBF_WRITE_FAILis used to indicate that the previous write failed, not that a historic write failed. The flag is cleared at buffer submission time - seexfs_buf_delwri_submit_buffers()andxfs_bwrite()- and so it is only set on buffers whose previous IO failed and hence is still dirty and has not been flushed back to disk.
Right, but the flag is only ever cleared in those particular submission paths (which doesn't include the resubmit path). The point being overlooked here is that it is never cleared on successful I/O completion. That means if an I/O fails, we set the flag and resubmit. If the resubmitted I/O succeeds, the flag remains set on the buffer until the next time it is written, but there's no guarantee the buffer is dirtied again before we unmount. Therefore, it's possible to have XBF_WRITE_FAIL set on clean/consistent buffers at unmount time without any underlying inconsistency.
Quoted 7 lines
If we hit this in
xfs_buftarg_wait()after we've pushed the AIL inxfs_log_quiesce()on unmount, then we've got write failures that could not be resolved by repeated retries, and the filesystem is, at this instant in time, inconsistent on disk.That's a shutdown error condition...
Yes, but XBF_WRITE_FAIL does not reflect that condition with certainty.
Quoted 30 lines
Quoted 6 lines
For that reason I've kind of looked
at this particular instance as more of a warning that I/O errors have occurred in the past and the user might want to verify it didn't result in unexpected damage. FWIW, I've observed plenty of these messages long after I've disabled error injection and allowed I/O to proceed and the fs to unmount cleanly.Right. THat's the whole point of the flag - the buffer has been dirtied and it hasn't been able to be written back when we are purging the buffer cache at the end of unmount. i.e. When
xfs_buftarg_wait()is called, all buffers should be clean because we are about to write an unmount record to mark the log clean once all the logged metadata is written back.What we do right now - write an unmount record after failing metadta writeback - is actually a bug, and that is one of the reasons why I suggested a shutdown should be done. i.e. we should not be writing an unmount record to mark the log clean if we failed to write back metadata. That metadata is still valid in the journal, and so it should remain valid in the journal to allow it to be replayed on the next mount. i.e. retry the writes from log recovery after the hardware failure has been addressed and the IO errors have gone away.
Tossing the dirty buffers unmount and then marking the journal clean is actually making the buffer write failure -worse-. Doing this guarantees the filesystem is inconsistent on disk (by updating the journal to indicate those writes actually succeeded) and absolutely requires
xfs_repairto fix as a result.
Quoted 9 lines
If we shut down on
XBF_WRITE_FAILbuffers inxfs_buftarg_wait(), we will not write an unmount record and so give the filesystem a chance to recover on next mount (e.g. after a power cycle to clear whatever raid hardware bug was being hit) and write that dirty metadata back without failure. If recovery fails with IO errors, then the user really does need to run repair. However, the situation at this point is still better than if we write a clean unmount record after write failures...
I agree with all of this when we are actually tossing dirty buffers. ;) See above that I'm referring to how this state can persist after error injection is disabled and I/O errors have ceased. Also note that the error handling code already shuts down the filesystem in the completion handling path of any buffer undergoing retries at unmount time (see XFS_MOUNT_UNMOUNTING). IIRC this was introduced to prevent hanging unmount on fs' configured with infinite retries, but it ensures the fs is shut down if dirty buffers have failed to write back before we get to xfs_wait_buftarg(). So there isn't a functional bug from what I can see if XBF_WRITE_FAIL does happen to reflect a failed+dirty buffer in xfs_wait_buftarg().
It's pretty clear that the implementation of XBF_WRITE_FAIL is unnecessarily fragile and confusing. Aside from not being able to establish the meaning of the xfs_wait_buftarg() warning message between the two of us (good luck, users ;P), I'm guessing it's also not obvious that there's an internal retry associated with each xfsaild submitted write, even for buffers that persistently fail. For example, xfsaild submits a buffer, I/O fails, we set XBF_WRITE_FAIL and resubmit, I/O fails again, XBF_WRITE_FAIL is already set and retries are configured so we unlock the buffer, xfsaild comes around again and requeues, delwri submit clears XBF_WRITE_FAIL and we start over from the beginning.
Perhaps a better solution would be to clear XBF_WRITE_FAIL on successful I/O completion. That way we only issue the internal retry once since the last time a buffer was successfully written and xfs_wait_buftarg() can legitimately warn about tossing dirty buffers and shut down. I do think it makes sense to shut down from there in that scenario as well, provided the logic is correct, simply because the dependency on error configuration shutdown behavior is not obvious. Thoughts?
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 6/13 xfs: fix duplicate verification from xfs_qm_dqflush()
In reply to Brian FosterThe pre-flush dquot verification in xfs_qm_dqflush() duplicates the read verifier by checking the dquot in the on-disk buffer. Instead, verify the in-core variant before it is flushed to the buffer.
--- fs/xfs/xfs_dquot.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
fs/xfs/xfs_dquot.c +4 -5
@@ -1116,13 +1116,12 @@ xfs_qm_dqflush( dqb = bp->b_addr + dqp->q_bufoffset; ddqp = &dqb->dd_diskdq; -/* -* A simple sanity check in case we got a corrupted dquot. -*/ - fa =xfs_dqblk_verify (mp, dqb,be32_to_cpu (ddqp->d_id),0 ); +/* sanity check the in-core structure before we flush */ + fa =xfs_dquot_verify (mp, &dqp->q_core,be32_to_cpu (dqp->q_core.d_id), +0 );if (fa) {xfs_alert (mp,"corrupt dquot ID 0x%x in memory at %pS" , -be32_to_cpu (ddqp->d_id), fa); +be32_to_cpu (dqp->q_core.d_id), fa);xfs_buf_relse (bp);xfs_dqfunlock (dqp);xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );
Signature · 1 line
2.21.1
Re: [PATCH v2 06/13] xfs: fix duplicate verification from xfs_qm_dqflush()
Quoted 5 lines · Brian Foster
The pre-flush dquot verification in
xfs_qm_dqflush()duplicates the read verifier by checking the dquot in the on-disk buffer. Instead, verify the in-core variant before it is flushed to the buffer.Signed-off-by Brian Foster <[email protected]>
Looks good.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 06/13] xfs: fix duplicate verification from xfs_qm_dqflush()
In reply to Brian FosterQuoted 5 lines · Brian Foster
The pre-flush dquot verification in
xfs_qm_dqflush()duplicates the read verifier by checking the dquot in the on-disk buffer. Instead, verify the in-core variant before it is flushed to the buffer.Signed-off-by Brian Foster <[email protected]>
Looks good,
PATCH v2 7/13 xfs: abort consistently on dquot flush failure
In reply to Brian FosterThe dquot flush handler effectively aborts the dquot flush if the filesystem is already shut down, but doesn't actually shut down if the flush fails. Update xfs_qm_dqflush() to consistently abort the dquot flush and shutdown the fs if the flush fails with an unexpected error.
--- fs/xfs/xfs_dquot.c | 32 ++++++++++---------------------- 1 file changed, 10 insertions(+), 22 deletions(-)
fs/xfs/xfs_dquot.c +10 -22
@@ -1068,6 +1068,7 @@ xfs_qm_dqflush( struct xfs_buf **bpp) {struct xfs_mount *mp = dqp->q_mount; +struct xfs_log_item *lip = &dqp->q_logitem.qli_item;struct xfs_buf *bp;struct xfs_dqblk *dqb;struct xfs_disk_dquot *ddqp;
@@ -1083,32 +1084,16 @@ xfs_qm_dqflush(xfs_qm_dqunpin_wait (dqp); -/* -* This may have been unpinned because the filesystem is shutting -* down forcibly. If that's the case we must not write this dquot -* to disk, because the log record didn't make it to disk. -* -* We also have to remove the log item from the AIL in this case, -* as we wait for an emptry AIL as part of the unmount process. -*/ -if (XFS_FORCED_SHUTDOWN (mp)) { -struct xfs_log_item *lip = &dqp->q_logitem.qli_item; - dqp->dq_flags &= ~XFS_DQ_DIRTY ; - -xfs_trans_ail_remove (lip,SHUTDOWN_CORRUPT_INCORE ); - - error = -EIO ; - goto out_unlock; - } -/* * Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , &bp, &xfs_dquot_buf_ops); -if (error) +if (error == -EAGAIN ) goto out_unlock; +if (error) + goto out_abort;/* * Calculate the location of the dquot inside the buffer.
@@ -1123,9 +1108,8 @@ xfs_qm_dqflush(xfs_alert (mp,"corrupt dquot ID 0x%x in memory at %pS" ,be32_to_cpu (dqp->q_core.d_id), fa);xfs_buf_relse (bp); -xfs_dqfunlock (dqp); -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); -return -EFSCORRUPTED ; + error = -EFSCORRUPTED ; + goto out_abort; }/* This is the only portion of data that needs to persist */
@@ -1174,6 +1158,10 @@ xfs_qm_dqflush( *bpp = bp;return 0 ; +out_abort: + dqp->dq_flags &= ~XFS_DQ_DIRTY ; +xfs_trans_ail_remove (lip,SHUTDOWN_CORRUPT_INCORE ); +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); out_unlock:xfs_dqfunlock (dqp);return error;
Signature · 1 line
2.21.1
Re: [PATCH v2 07/13] xfs: abort consistently on dquot flush failure
Quoted 5 lines · Brian Foster
The dquot flush handler effectively aborts the dquot flush if the filesystem is already shut down, but doesn't actually shut down if the flush fails. Update
xfs_qm_dqflush()to consistently abort the dquot flush and shutdown the fs if the flush fails with an unexpected error.
Looks good,
PATCH v2 8/13 xfs: elide the AIL lock on log item failure tracking
In reply to Brian FosterThe log item failed flag is used to indicate a log item was flushed but the underlying buffer was not successfully written to disk. If the error configuration allows for writeback retries, xfsaild uses the flag to resubmit the underlying buffer without acquiring the flush lock of the item.
The flag is currently set and cleared under the AIL lock and buffer lock in the ->iop_error() callback, invoked via ->b_iodone() at I/O completion time. The flag is checked at xfsaild push time under AIL lock and cleared under buffer lock before resubmission. If I/O eventually succeeds, the flag is cleared in the _done() handler for the associated item type, again under AIL lock and buffer lock.
As far as I can tell, the only reason for holding the AIL lock across sets/clears is to manage consistency between the log item bitop state and the temporary buffer pointer that is attached to the log item. The bit itself is used to manage consistency of the attached buffer, but is not enough to guarantee the buffer is still attached by the time xfsaild attempts to access it. However since failure state is always set or cleared under buffer lock (either via I/O completion or xfsaild), this particular case can be handled at item push time by verifying failure state once under buffer lock.
Remove the AIL lock protection from the various bits of log item failure state management and simplify the surrounding code where applicable. Use the buffer lock in the xfsaild resubmit code to detect failure state changes and temporarily treat the item as locked.
--- fs/xfs/xfs_buf_item.c | 4 ---- fs/xfs/xfs_dquot.c | 41 +++++++++++++++-------------------------- fs/xfs/xfs_inode_item.c | 11 +++-------- fs/xfs/xfs_trans_ail.c | 12 ++++++++++-- fs/xfs/xfs_trans_priv.h | 5 ----- 5 files changed, 28 insertions(+), 45 deletions(-)
fs/xfs/xfs_buf_item.c +0 -4
@@ -1038,7 +1038,6 @@ xfs_buf_do_callbacks_fail( struct xfs_buf *bp) {struct xfs_log_item *lip; -struct xfs_ail *ailp;/* * Buffer log item errors are handled directly by xfs_buf_item_push()
@@ -1050,13 +1049,10 @@ xfs_buf_do_callbacks_fail( lip = list_first_entry(&bp->b_li_list, struct xfs_log_item, li_bio_list); - ailp = lip->li_ailp; -spin_lock (&ailp->ail_lock);list_for_each_entry (lip, &bp->b_li_list, li_bio_list) {if (lip->li_ops->iop_error) lip->li_ops->iop_error (lip, bp); } -spin_unlock (&ailp->ail_lock); } static bool
fs/xfs/xfs_dquot.c +15 -26
49 lines not on this page · raw message
fs/xfs/xfs_inode_item.c +3 -8
@@ -682,9 +682,7 @@ xfs_iflush_done(* Scan the buffer IO completions for other inodes being completed and * attach them to the current inode log item. */ -list_add_tail (&lip->li_bio_list, &tmp); -list_for_each_entry_safe (blip, n, &bp->b_li_list, li_bio_list) {if (lip->li_cb != xfs_iflush_done)continue ;
@@ -695,15 +693,13 @@ xfs_iflush_done(* the AIL lock. */ iip =INODE_ITEM (blip); -if ((iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) || -test_bit (XFS_LI_FAILED , &blip->li_flags)) +if (iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) need_ail++; }/* make sure we capture the state of the initial inode. */ iip =INODE_ITEM (lip); -if ((iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) || -test_bit (XFS_LI_FAILED , &lip->li_flags)) +if (iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) need_ail++;/*
@@ -732,8 +728,6 @@ xfs_iflush_done(xfs_lsn_t lsn =xfs_ail_delete_one (ailp, blip);if (!tail_lsn && lsn) tail_lsn = lsn; - }else { -xfs_clear_li_failed (blip); } }xfs_ail_update_finish (ailp, tail_lsn);
@@ -746,6 +740,7 @@ xfs_iflush_done(*/ list_for_each_entry_safe (blip, n, &tmp, li_bio_list) {list_del_init (&blip->li_bio_list); +xfs_clear_li_failed (blip); iip =INODE_ITEM (blip); iip->ili_logged =0 ; iip->ili_last_fields =0 ;
fs/xfs/xfs_trans_ail.c +10 -2
@@ -368,15 +368,23 @@ xfsaild_resubmit_item( {struct xfs_buf *bp = lip->li_buf; -if (!xfs_buf_trylock (bp)) +/* +* Log item state bits are racy so we cannot assume the temporary buffer +* pointer is set. Treat the item as locked if the pointer is clear or +* the failure state has changed once we've locked out I/O completion. +*/ +if (!bp || !xfs_buf_trylock (bp))return XFS_ITEM_LOCKED ; +if (!test_bit (XFS_LI_FAILED , &lip->li_flags)) { +xfs_buf_unlock (bp); +return XFS_ITEM_LOCKED ; + }if (!xfs_buf_delwri_queue (bp, buffer_list)) {xfs_buf_unlock (bp);return XFS_ITEM_FLUSHING ; } -/* protected by ail_lock */ list_for_each_entry (lip, &bp->b_li_list, li_bio_list)xfs_clear_li_failed (lip);
fs/xfs/xfs_trans_priv.h +0 -5
@@ -158,9 +158,6 @@ xfs_clear_li_failed( {struct xfs_buf *bp = lip->li_buf; -ASSERT (test_bit (XFS_LI_IN_AIL , &lip->li_flags)); -lockdep_assert_held (&lip->li_ailp->ail_lock); -if (test_and_clear_bit (XFS_LI_FAILED , &lip->li_flags)) { lip->li_buf =NULL ;xfs_buf_rele (bp);
@@ -172,8 +169,6 @@ xfs_set_li_failed( struct xfs_log_item *lip, struct xfs_buf *bp) { - lockdep_assert_held(&lip->li_ailp->ail_lock); -if (!test_and_set_bit (XFS_LI_FAILED , &lip->li_flags)) {xfs_buf_hold (bp); lip->li_buf = bp;
Signature · 1 line
2.21.1
Re: [PATCH v2 08/13] xfs: elide the AIL lock on log item failure tracking
Quoted 6 lines · Brian Foster
The log item failed flag is used to indicate a log item was flushed but the underlying buffer was not successfully written to disk. If the error configuration allows for writeback retries, xfsaild uses the flag to resubmit the underlying buffer without acquiring the flush lock of the item.
Quoted 6 lines
The flag is currently set and cleared under the AIL lock and buffer lock in the ->iop_error() callback, invoked via ->b_iodone() at I/O completion time. The flag is checked at xfsaild push time under AIL lock and cleared under buffer lock before resubmission. If I/O eventually succeeds, the flag is cleared in the
_done()handler for the associated item type, again under AIL lock and buffer lock.
Actually, I think you've missed the relevant lock here: the item's flush lock. The XFS_LI_FAILED flag is item flush state, and flush state is protected by the flush lock. When the item has been flushed and attached to the buffer for completion callbacks, the flush lock context gets handed to the buffer.
i.e. the buffer owns the flush lock and so while that buffer is locked for IO we know that the item flush state (and hence the XFS_LI_FAILED flag) is exclusively owned by the holder of the buffer lock.
(Note: this is how xfs_ifree_cluster() works - it grabs the buffer lock then walks the items on the buffer and changes the callback functions because those items are flush locked and hence holding the buffer lock gives exclusive access to the flush state of those items....)
Quoted 6 lines
As far as I can tell, the only reason for holding the AIL lock across sets/clears is to manage consistency between the log item bitop state and the temporary buffer pointer that is attached to the log item. The bit itself is used to manage consistency of the attached buffer, but is not enough to guarantee the buffer is still attached by the time xfsaild attempts to access it.
Correct. The guarantee that the buffer is still attached to the log item is what the AIL lock provides us with.
Quoted 4 lines
However since
failure state is always set or cleared under buffer lock (either via I/O completion or xfsaild), this particular case can be handled at item push time by verifying failure state once under buffer lock.
In theory, yes, but there's a problem before you get that buffer lock. That is: what serialises access to lip->li_buf?
The xfsaild does not hold a reference to the buffer and, without the AIL lock to provide synchronisation, the log item reference to the buffer can be dropped asynchronously by buffer IO completion. Hence the buffer could be freed between the xfsaild reading lip->li_buf and calling xfs_buf_trylock(bp). i.e. this introduces a use-after-free race condition on the initial buffer access.
IOWs, the xfsaild cannot access lip->li_buf safely unless the set/clear operations are protected by the same lock the xfsaild holds. The xfsaild holds neither the buffer lock, a buffer reference or an item flush lock, hence it's the AIL lock or nothing...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 08/13] xfs: elide the AIL lock on log item failure tracking
Quoted 20 lines · Dave Chinner
Quoted 6 lines · Brian Foster
The log item failed flag is used to indicate a log item was flushed but the underlying buffer was not successfully written to disk. If the error configuration allows for writeback retries, xfsaild uses the flag to resubmit the underlying buffer without acquiring the flush lock of the item.
Quoted 6 lines
The flag is currently set and cleared under the AIL lock and buffer lock in the ->iop_error() callback, invoked via ->b_iodone() at I/O completion time. The flag is checked at xfsaild push time under AIL lock and cleared under buffer lock before resubmission. If I/O eventually succeeds, the flag is cleared in the
_done()handler for the associated item type, again under AIL lock and buffer lock.Actually, I think you've missed the relevant lock here: the item's flush lock. The
XFS_LI_FAILEDflag is item flush state, and flush state is protected by the flush lock. When the item has been flushed and attached to the buffer for completion callbacks, the flush lock context gets handed to the buffer.
Quoted 5 lines
i.e. the buffer owns the flush lock and so while that buffer is locked for IO we know that the item flush state (and hence the
XFS_LI_FAILEDflag) is exclusively owned by the holder of the buffer lock.
Right..
Quoted 16 lines
(Note: this is how
xfs_ifree_cluster()works - it grabs the buffer lock then walks the items on the buffer and changes the callback functions because those items are flush locked and hence holding the buffer lock gives exclusive access to the flush state of those items....)Quoted 6 lines
As far as I can tell, the only reason for holding the AIL lock across sets/clears is to manage consistency between the log item bitop state and the temporary buffer pointer that is attached to the log item. The bit itself is used to manage consistency of the attached buffer, but is not enough to guarantee the buffer is still attached by the time xfsaild attempts to access it.
Correct. The guarantee that the buffer is still attached to the log item is what the AIL lock provides us with.
Quoted 8 lines
Quoted 4 lines
However since
failure state is always set or cleared under buffer lock (either via I/O completion or xfsaild), this particular case can be handled at item push time by verifying failure state once under buffer lock.In theory, yes, but there's a problem before you get that buffer lock. That is: what serialises access to
lip->li_buf?
That's partly what I was referring to above by the push time changes. This patch was an attempt to replace reliance on ail_lock with a push time sequence that would serialize access to a failed buffer (essentially relying on the failed bit). Whether it's correct or not is another matter... ;)
Quoted 7 lines
The xfsaild does not hold a reference to the buffer and, without the AIL lock to provide synchronisation, the log item reference to the buffer can be dropped asynchronously by buffer IO completion. Hence the buffer could be freed between the xfsaild reading
lip->li_bufand callingxfs_buf_trylock(bp). i.e. this introduces a use-after-free race condition on the initial buffer access.
Hmm.. the log item holds a temporary reference to the buffer when the failed state is set. On submission, xfsaild queues the failed buffer (new ref for the delwri queue), clears the failed state and drops the failure reference of every failed item that is attached. xfsaild is also the only task that knows how to process LI_FAILED items, so I don't think we'd ever race with a failed buffer becoming "unfailed" from xfsaild (which is the scenario where a buffer could disappear from I/O completion). In some sense, clearing LI_FAILED of an item is essentially retaking ownership of the item flush lock and hold of the underlying buffer, but the existing code is certainly not written that way.
The issue I was wrestling with wrt to this patch was primarily maintaining consistency between the bit and the assignment of the pointer on a failing I/O. E.g., the buffer pointer is set by the task that sets the bit, but xfsaild checking the bit doesn't guarantee the pointer had been set yet. I did add a post-buffer lock LI_FAILED check as well, but that probably could have been an assert.
Quoted 5 lines
IOWs, the xfsaild cannot access
lip->li_bufsafely unless the set/clear operations are protected by the same lock the xfsaild holds. The xfsaild holds neither the buffer lock, a buffer reference or an item flush lock, hence it's the AIL lock or nothing...
Yeah, that was my impression from reading the code. I realize from this discussion that this doesn't actually simplify the logic. That was the primary goal, so I think I need to revisit the approach here. Even if this is correct (which I'm still not totally sure of), it's fragile in the sense that it partly relies on single-threadedness of xfsaild. I initially thought about adding a log item spinlock for this kind of thing, but it didn't (and still doesn't) seem appropriate to burden every log item in the system with a spinlock for the rare case of buffer I/O errors.
I mentioned in an earlier patch that it would be nice to consider removing ->li_buf entirely but hadn't thought it through. That might alleviate the need to serialize the pointer and the state in the first place as well as remove the subtle ordering requirement from the resubmit code to manage the buffer reference. Suppose we relied on LI_FAILED to represent the buffer hold + flush lock, and then relied on the hold to facilitate an in-core lookup from xfsaild. For example:
xfs_set_li_failed(lip, bp)
{
/* keep the backing buffer in-core so long as the item is failed */
if (!test_and_set_bit(XFS_LI_FAILED, &lip->li_flags))
xfs_buf_hold(bp);
}xfs_clear_li_failed(lip, bp)
{
if (test_and_clear_bit(XFS_LI_FAILED, &lip->li_flags))
xfs_buf_rele(bp);
}/*
* helper to get a mapping from a buffer backed log item and use it to
* find an in-core buf
*/xfs_item_to_bp(lip)
{
if (inode item) {
...
blk = ip->imap->im_blkno;
len = ip->imap->im_len;
} else if (dquot item) {
...
blk = dqp->q_blkno;
len = mp->m_quotainfo->qi_dqchunklen;
}
return xfs_buf_incore(blk, len, XBF_TRYLOCK);
}xfsaild_resubmit_item(lip)
{
...
bp = xfs_item_to_bp(lip);
if (!bp)
return XFS_ITEM_LOCKED;
if (!test_bit(XFS_LI_FAILED, &lip->li_flags))
goto out_unlock;
/* drop all failure refs */
list_for_each_entry(lip, &bp->b_li_list, li_bio_list)
xfs_clear_li_failed(lip, bp);
xfs_buf_delwri_queue(...);
xfs_buf_relse(bp);
return XFS_ITEM_SUCCESS;
}I think that would push everything under the buffer lock (and remove ->li_buf) with the caveat that we'd have to lock the inode/dquot to get the buffer. Any thoughts on something like that? Note that at this point I might drop this patch from the series regardless due to the complexity mismatch and consider it separately.
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 08/13] xfs: elide the AIL lock on log item failure tracking
Quoted 68 lines · Brian Foster
Quoted 20 lines · Dave Chinner
Quoted 6 lines · Brian Foster
The log item failed flag is used to indicate a log item was flushed but the underlying buffer was not successfully written to disk. If the error configuration allows for writeback retries, xfsaild uses the flag to resubmit the underlying buffer without acquiring the flush lock of the item.
Quoted 6 lines
The flag is currently set and cleared under the AIL lock and buffer lock in the ->iop_error() callback, invoked via ->b_iodone() at I/O completion time. The flag is checked at xfsaild push time under AIL lock and cleared under buffer lock before resubmission. If I/O eventually succeeds, the flag is cleared in the
_done()handler for the associated item type, again under AIL lock and buffer lock.Actually, I think you've missed the relevant lock here: the item's flush lock. The
XFS_LI_FAILEDflag is item flush state, and flush state is protected by the flush lock. When the item has been flushed and attached to the buffer for completion callbacks, the flush lock context gets handed to the buffer.Quoted 5 lines
i.e. the buffer owns the flush lock and so while that buffer is locked for IO we know that the item flush state (and hence the
XFS_LI_FAILEDflag) is exclusively owned by the holder of the buffer lock.Right..
Quoted 16 lines
(Note: this is how
xfs_ifree_cluster()works - it grabs the buffer lock then walks the items on the buffer and changes the callback functions because those items are flush locked and hence holding the buffer lock gives exclusive access to the flush state of those items....)Quoted 6 lines
As far as I can tell, the only reason for holding the AIL lock across sets/clears is to manage consistency between the log item bitop state and the temporary buffer pointer that is attached to the log item. The bit itself is used to manage consistency of the attached buffer, but is not enough to guarantee the buffer is still attached by the time xfsaild attempts to access it.
Correct. The guarantee that the buffer is still attached to the log item is what the AIL lock provides us with.
Quoted 8 lines
Quoted 4 lines
However since
failure state is always set or cleared under buffer lock (either via I/O completion or xfsaild), this particular case can be handled at item push time by verifying failure state once under buffer lock.In theory, yes, but there's a problem before you get that buffer lock. That is: what serialises access to
lip->li_buf?That's partly what I was referring to above by the push time changes. This patch was an attempt to replace reliance on
ail_lockwith a push time sequence that would serialize access to a failed buffer (essentially relying on the failed bit). Whether it's correct or not is another matter... ;)Quoted 7 lines
The xfsaild does not hold a reference to the buffer and, without the AIL lock to provide synchronisation, the log item reference to the buffer can be dropped asynchronously by buffer IO completion. Hence the buffer could be freed between the xfsaild reading
lip->li_bufand callingxfs_buf_trylock(bp). i.e. this introduces a use-after-free race condition on the initial buffer access.
Quoted 8 lines
Hmm.. the log item holds a temporary reference to the buffer when the failed state is set. On submission, xfsaild queues the failed buffer (new ref for the delwri queue), clears the failed state and drops the failure reference of every failed item that is attached. xfsaild is also the only task that knows how to process
LI_FAILEDitems, so I don't think we'd ever race with a failed buffer becoming "unfailed" from xfsaild (which is the scenario where a buffer could disappear from I/O completion).
Sure we can. every inode on the buffer has XFS_LI_FAILED set on it, so the first inode in the AIL triggers the resubmit and starts the IO. the AIL runs again, coming across another inode on the buffer further into the AIL. That inode has been modified in memory while the flush was in progress, so it no longer gets removed from the AIL by the IO completion. Instead, xfs_clear_li_failed() is called from xfs_iflush_done() and the buffer is removed from the logitem and the reference dropped.
Quoted 3 lines
In some sense, clearing
LI_FAILEDof an item is essentially retaking ownership of the item flush lock and hold of the underlying buffer, but the existing code is certainly not written that way.
Only IO completion clears the LI_FAILED state of the item, not the xfsaild. IO completion already owns the flush lock - the xfsaild only holds it long enough to flush an inode to the buffer and then give the flush lock to the buffer.
Also, we clear LI_FAILED when removing the item from the AIL under the AIL lock, so the only case here that we are concerned about is the above "inode was relogged while being flushed" situation. THis situation is rare, failed buffers are rare, and so requiring the AIL lock to be held is a performance limiting factor here...
Quoted 6 lines
The issue I was wrestling with wrt to this patch was primarily maintaining consistency between the bit and the assignment of the pointer on a failing I/O. E.g., the buffer pointer is set by the task that sets the bit, but xfsaild checking the bit doesn't guarantee the pointer had been set yet. I did add a post-buffer lock
LI_FAILEDcheck as well, but that probably could have been an assert.
SO you've been focussing on races that may occur when the setting of the li_buf, but I'm looking at races involving the clearing of the li_buf pointer. :)
Quoted 6 lines
Quoted 5 lines
IOWs, the xfsaild cannot access
lip->li_bufsafely unless the set/clear operations are protected by the same lock the xfsaild holds. The xfsaild holds neither the buffer lock, a buffer reference or an item flush lock, hence it's the AIL lock or nothing...
Quoted 9 lines
Yeah, that was my impression from reading the code. I realize from this discussion that this doesn't actually simplify the logic. That was the primary goal, so I think I need to revisit the approach here. Even if this is correct (which I'm still not totally sure of), it's fragile in the sense that it partly relies on single-threadedness of xfsaild. I initially thought about adding a log item spinlock for this kind of thing, but it didn't (and still doesn't) seem appropriate to burden every log item in the system with a spinlock for the rare case of buffer I/O errors.
I feel this is all largely a moot, because my patches result in this this whole problem of setting/clearing the li_buf for failed buffers go away altogether. Hence I would suggest that this patch gets dropped for now, because getting rid of the AIL lock is much less troublesome once LI_FAILED is no long dependent on the inode/dquot log item taking temporary references to the underlying buffer....
Quoted 3 lines
I mentioned in an earlier patch that it would be nice to consider removing ->li_buf entirely but hadn't thought it through.
I'm going the opposite way entirely, such that LI_FAILED doesn't have any special state associated with it at all...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 08/13] xfs: elide the AIL lock on log item failure tracking
Quoted 78 lines · Dave Chinner
Quoted 68 lines · Brian Foster
Quoted 20 lines · Dave Chinner
Quoted 6 lines · Brian Foster
The log item failed flag is used to indicate a log item was flushed but the underlying buffer was not successfully written to disk. If the error configuration allows for writeback retries, xfsaild uses the flag to resubmit the underlying buffer without acquiring the flush lock of the item.
Quoted 6 lines
The flag is currently set and cleared under the AIL lock and buffer lock in the ->iop_error() callback, invoked via ->b_iodone() at I/O completion time. The flag is checked at xfsaild push time under AIL lock and cleared under buffer lock before resubmission. If I/O eventually succeeds, the flag is cleared in the
_done()handler for the associated item type, again under AIL lock and buffer lock.Actually, I think you've missed the relevant lock here: the item's flush lock. The
XFS_LI_FAILEDflag is item flush state, and flush state is protected by the flush lock. When the item has been flushed and attached to the buffer for completion callbacks, the flush lock context gets handed to the buffer.Quoted 5 lines
i.e. the buffer owns the flush lock and so while that buffer is locked for IO we know that the item flush state (and hence the
XFS_LI_FAILEDflag) is exclusively owned by the holder of the buffer lock.Right..
Quoted 16 lines
(Note: this is how
xfs_ifree_cluster()works - it grabs the buffer lock then walks the items on the buffer and changes the callback functions because those items are flush locked and hence holding the buffer lock gives exclusive access to the flush state of those items....)Quoted 6 lines
As far as I can tell, the only reason for holding the AIL lock across sets/clears is to manage consistency between the log item bitop state and the temporary buffer pointer that is attached to the log item. The bit itself is used to manage consistency of the attached buffer, but is not enough to guarantee the buffer is still attached by the time xfsaild attempts to access it.
Correct. The guarantee that the buffer is still attached to the log item is what the AIL lock provides us with.
Quoted 8 lines
Quoted 4 lines
However since
failure state is always set or cleared under buffer lock (either via I/O completion or xfsaild), this particular case can be handled at item push time by verifying failure state once under buffer lock.In theory, yes, but there's a problem before you get that buffer lock. That is: what serialises access to
lip->li_buf?That's partly what I was referring to above by the push time changes. This patch was an attempt to replace reliance on
ail_lockwith a push time sequence that would serialize access to a failed buffer (essentially relying on the failed bit). Whether it's correct or not is another matter... ;)Quoted 7 lines
The xfsaild does not hold a reference to the buffer and, without the AIL lock to provide synchronisation, the log item reference to the buffer can be dropped asynchronously by buffer IO completion. Hence the buffer could be freed between the xfsaild reading
lip->li_bufand callingxfs_buf_trylock(bp). i.e. this introduces a use-after-free race condition on the initial buffer access.Quoted 8 lines
Hmm.. the log item holds a temporary reference to the buffer when the failed state is set. On submission, xfsaild queues the failed buffer (new ref for the delwri queue), clears the failed state and drops the failure reference of every failed item that is attached. xfsaild is also the only task that knows how to process
LI_FAILEDitems, so I don't think we'd ever race with a failed buffer becoming "unfailed" from xfsaild (which is the scenario where a buffer could disappear from I/O completion).
Quoted 9 lines
Sure we can. every inode on the buffer has
XFS_LI_FAILEDset on it, so the first inode in the AIL triggers the resubmit and starts the IO. the AIL runs again, coming across another inode on the buffer further into the AIL. That inode has been modified in memory while the flush was in progress, so it no longer gets removed from the AIL by the IO completion. Instead,xfs_clear_li_failed()is called fromxfs_iflush_done()and the buffer is removed from the logitem and the reference dropped.
When xfsaild encounters the first failed item, the buffer resubmit path (xfsaild_resubmit_item(), as of this series) calls xfs_clear_li_failed() on every inode attached to the buffer. AFAICT, that means xfsaild will see any associated inode as FLUSHING (or PINNED, if another in-core modification has come through) once the buffer is requeued.
Quoted 9 lines
Quoted 3 lines
In some sense, clearing
LI_FAILEDof an item is essentially retaking ownership of the item flush lock and hold of the underlying buffer, but the existing code is certainly not written that way.Only IO completion clears the
LI_FAILEDstate of the item, not the xfsaild. IO completion already owns the flush lock - the xfsaild only holds it long enough to flush an inode to the buffer and then give the flush lock to the buffer.
Quoted 6 lines
Also, we clear
LI_FAILEDwhen removing the item from the AIL under the AIL lock, so the only case here that we are concerned about is the above "inode was relogged while being flushed" situation. THis situation is rare, failed buffers are rare, and so requiring the AIL lock to be held is a performance limiting factor here...
Yep, though because of the above I think I/O completion clearing the failed state might require a more subtle dance between xfsaild and another submission context, such as if reclaim happens to flush an inode that wasn't already attached to a previously failed buffer (for example).
Eh, it doesn't matter that much in any event because..
Quoted 27 lines
Quoted 6 lines
The issue I was wrestling with wrt to this patch was primarily maintaining consistency between the bit and the assignment of the pointer on a failing I/O. E.g., the buffer pointer is set by the task that sets the bit, but xfsaild checking the bit doesn't guarantee the pointer had been set yet. I did add a post-buffer lock
LI_FAILEDcheck as well, but that probably could have been an assert.SO you've been focussing on races that may occur when the setting of the
li_buf, but I'm looking at races involving the clearing of theli_bufpointer. :)Quoted 6 lines
Quoted 5 lines
IOWs, the xfsaild cannot access
lip->li_bufsafely unless the set/clear operations are protected by the same lock the xfsaild holds. The xfsaild holds neither the buffer lock, a buffer reference or an item flush lock, hence it's the AIL lock or nothing...Quoted 9 lines
Yeah, that was my impression from reading the code. I realize from this discussion that this doesn't actually simplify the logic. That was the primary goal, so I think I need to revisit the approach here. Even if this is correct (which I'm still not totally sure of), it's fragile in the sense that it partly relies on single-threadedness of xfsaild. I initially thought about adding a log item spinlock for this kind of thing, but it didn't (and still doesn't) seem appropriate to burden every log item in the system with a spinlock for the rare case of buffer I/O errors.
Quoted 7 lines
I feel this is all largely a moot, because my patches result in this this whole problem of setting/clearing the
li_buffor failed buffers go away altogether. Hence I would suggest that this patch gets dropped for now, because getting rid of the AIL lock is much less troublesome onceLI_FAILEDis no long dependent on the inode/dquot log item taking temporary references to the underlying buffer....
... I'm good with that. ;) This was intended to be a low level cleanup to eliminate a fairly minor ail_lock abuse. Working around the ->li_buf thing is the primary challenge, so if you have a broader rework that addresses that as a side effect then that presumably makes things easier. I'll revisit this if necessary once your work is further along...
Brian
Quoted 14 lines
Quoted 3 lines
I mentioned in an earlier patch that it would be nice to consider removing ->li_buf entirely but hadn't thought it through.
I'm going the opposite way entirely, such that
LI_FAILEDdoesn't have any special state associated with it at all...Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 9/13 xfs: clean up AIL log item removal functions
In reply to Brian FosterWe have two AIL removal functions with slightly different semantics. xfs_trans_ail_delete() expects the caller to have the AIL lock and for the associated item to be AIL resident. If not, the filesystem is shut down. xfs_trans_ail_remove() acquires the AIL lock, checks that the item is AIL resident and calls the former if so.
These semantics lead to confused usage between the two. For example, the _remove() variant takes a shutdown parameter to pass to the _delete() variant, but immediately returns if the AIL bit is not set. This means that _remove() would never shut down if an item is not AIL resident, even though it appears that many callers would expect it to.
Make the following changes to clean up both of these functions:
- Most callers of xfs_trans_ail_delete() acquire the AIL lock just before the call. Update _delete() to acquire the lock and open code the couple of callers that make additional checks under AIL lock.
- Drop the unnecessary ailp parameter from _delete().
- Drop the unused shutdown parameter from _remove() and open code the implementation.
In summary, this leaves a _delete() variant that expects an AIL resident item and a _remove() helper that checks the AIL bit. Audit the existing callsites for use of the appropriate function and update as necessary.
--- fs/xfs/xfs_bmap_item.c | 2 +- fs/xfs/xfs_buf_item.c | 21 ++++++++------------- fs/xfs/xfs_dquot.c | 12 +++++++----- fs/xfs/xfs_dquot_item.c | 2 +- fs/xfs/xfs_extfree_item.c | 2 +- fs/xfs/xfs_inode_item.c | 6 +----- fs/xfs/xfs_refcount_item.c | 2 +- fs/xfs/xfs_rmap_item.c | 2 +- fs/xfs/xfs_trans_ail.c | 3 ++- fs/xfs/xfs_trans_priv.h | 17 +++++++++-------- 10 files changed, 32 insertions(+), 37 deletions(-)
fs/xfs/xfs_bmap_item.c +1 -1
@@ -51,7 +51,7 @@ xfs_bui_release( {ASSERT (atomic_read (&buip->bui_refcount) >0 );if (atomic_dec_and_test (&buip->bui_refcount)) { -xfs_trans_ail_remove (&buip->bui_item,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (&buip->bui_item,SHUTDOWN_LOG_IO_ERROR );xfs_bui_item_free (buip); } }
fs/xfs/xfs_buf_item.c +8 -13
@@ -410,7 +410,6 @@ xfs_buf_item_unpin( {struct xfs_buf_log_item *bip =BUF_ITEM (lip);xfs_buf_t *bp = bip->bli_buf; -struct xfs_ail *ailp = lip->li_ailp;int stale = bip->bli_flags &XFS_BLI_STALE ;int freed;
@@ -463,8 +462,7 @@ xfs_buf_item_unpin(list_del_init (&bp->b_li_list); bp->b_iodone =NULL ; }else { -spin_lock (&ailp->ail_lock); -xfs_trans_ail_delete (ailp, lip,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bp);ASSERT (bp->b_log_item ==NULL ); }
@@ -560,7 +558,7 @@ xfs_buf_item_put(* state. */ if (aborted) -xfs_trans_ail_remove (lip,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bip->bli_buf);return true; }
@@ -1201,22 +1199,19 @@ xfs_buf_iodone(struct xfs_buf *bp, struct xfs_log_item *lip) { - struct xfs_ail *ailp = lip->li_ailp; - ASSERT(BUF_ITEM(lip)->bli_buf == bp);xfs_buf_rele (bp);/* -* If we are forcibly shutting down, this may well be -* off the AIL already. That's because we simulate the -* log-committed callbacks to unpin these buffers. Or we may never -* have put this item on AIL because of the transaction was -* aborted forcibly. xfs_trans_ail_delete() takes care of these. +* If we are forcibly shutting down, this may well be off the AIL +* already. That's because we simulate the log-committed callbacks to +* unpin these buffers. Or we may never have put this item on AIL +* because of the transaction was aborted forcibly. +* xfs_trans_ail_delete() takes care of these. * * Either way, AIL is useless if we're forcing a shutdown. */ -spin_lock (&ailp->ail_lock); -xfs_trans_ail_delete (ailp, lip,SHUTDOWN_CORRUPT_INCORE ); +xfs_trans_ail_delete (lip,SHUTDOWN_CORRUPT_INCORE );xfs_buf_item_free (BUF_ITEM (lip)); }
fs/xfs/xfs_dquot.c +7 -5
@@ -1021,6 +1021,7 @@ xfs_qm_dqflush_done(struct xfs_dq_logitem *qip = (struct xfs_dq_logitem *)lip;struct xfs_dquot *dqp = qip->qli_dquot;struct xfs_ail *ailp = lip->li_ailp; +xfs_lsn_t tail_lsn;/* * Only pull the item from the AIL if its location in the log has not
@@ -1032,10 +1033,11 @@ xfs_qm_dqflush_done( goto out;spin_lock (&ailp->ail_lock); -if (lip->li_lsn == qip->qli_flush_lsn) -/* xfs_trans_ail_delete() drops the AIL lock */ -xfs_trans_ail_delete (ailp, lip,SHUTDOWN_CORRUPT_INCORE ); -else +if (lip->li_lsn == qip->qli_flush_lsn) { +/* xfs_ail_update_finish() drops the AIL lock */ + tail_lsn =xfs_ail_delete_one (ailp, lip); +xfs_ail_update_finish (ailp, tail_lsn); + }else spin_unlock (&ailp->ail_lock); out:
@@ -1149,7 +1151,7 @@ xfs_qm_dqflush( out_abort: dqp->dq_flags &= ~XFS_DQ_DIRTY ; -xfs_trans_ail_remove (lip,SHUTDOWN_CORRUPT_INCORE ); +xfs_trans_ail_remove (lip);xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); out_unlock:xfs_dqfunlock (dqp);
fs/xfs/xfs_dquot_item.c +1 -1
@@ -343,7 +343,7 @@ xfs_qm_qoff_logitem_relse(ASSERT (test_bit (XFS_LI_IN_AIL , &lip->li_flags) ||test_bit (XFS_LI_ABORTED , &lip->li_flags) ||XFS_FORCED_SHUTDOWN (lip->li_mountp)); -xfs_trans_ail_remove (lip,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_remove (lip);kmem_free (lip->li_lv_shadow);kmem_free (qoff); }
fs/xfs/xfs_extfree_item.c +1 -1
@@ -55,7 +55,7 @@ xfs_efi_release( {ASSERT (atomic_read (&efip->efi_refcount) >0 );if (atomic_dec_and_test (&efip->efi_refcount)) { -xfs_trans_ail_remove (&efip->efi_item,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (&efip->efi_item,SHUTDOWN_LOG_IO_ERROR );xfs_efi_item_free (efip); } }
fs/xfs/xfs_inode_item.c +1 -5
@@ -763,11 +763,7 @@ xfs_iflush_abort(xfs_inode_log_item_t *iip = ip->i_itemp;if (iip) { -if (test_bit (XFS_LI_IN_AIL , &iip->ili_item.li_flags)) { -xfs_trans_ail_remove (&iip->ili_item, - stale ?SHUTDOWN_LOG_IO_ERROR : -SHUTDOWN_CORRUPT_INCORE ); - } +xfs_trans_ail_remove (&iip->ili_item); iip->ili_logged =0 ;/* * Clear the ili_last_fields bits now that we know that the
fs/xfs/xfs_refcount_item.c +1 -1
@@ -50,7 +50,7 @@ xfs_cui_release( {ASSERT (atomic_read (&cuip->cui_refcount) >0 );if (atomic_dec_and_test (&cuip->cui_refcount)) { -xfs_trans_ail_remove (&cuip->cui_item,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (&cuip->cui_item,SHUTDOWN_LOG_IO_ERROR );xfs_cui_item_free (cuip); } }
fs/xfs/xfs_rmap_item.c +1 -1
@@ -50,7 +50,7 @@ xfs_rui_release( {ASSERT (atomic_read (&ruip->rui_refcount) >0 );if (atomic_dec_and_test (&ruip->rui_refcount)) { -xfs_trans_ail_remove (&ruip->rui_item,SHUTDOWN_LOG_IO_ERROR ); +xfs_trans_ail_delete (&ruip->rui_item,SHUTDOWN_LOG_IO_ERROR );xfs_rui_item_free (ruip); } }
fs/xfs/xfs_trans_ail.c +2 -1
@@ -872,13 +872,14 @@ xfs_ail_delete_one(*/ void xfs_trans_ail_delete ( -struct xfs_ail *ailp,struct xfs_log_item *lip,int shutdown_type) { +struct xfs_ail *ailp = lip->li_ailp;struct xfs_mount *mp = ailp->ail_mount;xfs_lsn_t tail_lsn; +spin_lock (&ailp->ail_lock);if (!test_bit (XFS_LI_IN_AIL , &lip->li_flags)) {spin_unlock (&ailp->ail_lock);if (!XFS_FORCED_SHUTDOWN (mp)) {
fs/xfs/xfs_trans_priv.h +9 -8
@@ -94,22 +94,23 @@ xfs_trans_ail_update(xfs_lsn_t xfs_ail_delete_one (struct xfs_ail *ailp,struct xfs_log_item *lip); void xfs_ail_update_finish(struct xfs_ail *ailp, xfs_lsn_t old_lsn) __releases(ailp->ail_lock); -void xfs_trans_ail_delete (struct xfs_ail *ailp,struct xfs_log_item *lip, -int shutdown_type); +void xfs_trans_ail_delete (struct xfs_log_item *lip,int shutdown_type);static inline void xfs_trans_ail_remove ( -struct xfs_log_item *lip, -int shutdown_type) +struct xfs_log_item *lip) {struct xfs_ail *ailp = lip->li_ailp; +xfs_lsn_t tail_lsn;spin_lock (&ailp->ail_lock); -/* xfs_trans_ail_delete() drops the AIL lock */ -if (test_bit (XFS_LI_IN_AIL , &lip->li_flags)) -xfs_trans_ail_delete (ailp, lip, shutdown_type); -else +/* xfs_ail_update_finish() drops the AIL lock */ +if (test_bit (XFS_LI_IN_AIL , &lip->li_flags)) { + tail_lsn =xfs_ail_delete_one (ailp, lip); +xfs_ail_update_finish (ailp, tail_lsn); + }else {spin_unlock (&ailp->ail_lock); + } }void xfs_ail_push (struct xfs_ail *,xfs_lsn_t );
Signature · 1 line
2.21.1
Re: [PATCH v2 09/13] xfs: clean up AIL log item removal functions
Quoted 23 lines · Brian Foster
We have two AIL removal functions with slightly different semantics.
xfs_trans_ail_delete()expects the caller to have the AIL lock and for the associated item to be AIL resident. If not, the filesystem is shut down.xfs_trans_ail_remove()acquires the AIL lock, checks that the item is AIL resident and calls the former if so.These semantics lead to confused usage between the two. For example, the
_remove()variant takes a shutdown parameter to pass to the_delete()variant, but immediately returns if the AIL bit is not set. This means that_remove()would never shut down if an item is not AIL resident, even though it appears that many callers would expect it to.Make the following changes to clean up both of these functions:
- Most callers of
xfs_trans_ail_delete()acquire the AIL lock just before the call. Update_delete()to acquire the lock and open code the couple of callers that make additional checks under AIL lock.
- Drop the unnecessary ailp parameter from_delete().
- Drop the unused shutdown parameter from_remove()and open code the implementation.
Quoted 6 lines
In summary, this leaves a
_delete()variant that expects an AIL resident item and a_remove()helper that checks the AIL bit. Audit the existing callsites for use of the appropriate function and update as necessary.Signed-off-by Brian Foster <[email protected]>
In conjunction with the followed patch to combine the remove and delete functions, I'm good with this.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 09/13] xfs: clean up AIL log item removal functions
In reply to Brian FosterQuoted 10 lines · Brian Foster
Make the following changes to clean up both of these functions:
- Most callers of
xfs_trans_ail_delete()acquire the AIL lock just before the call. Update_delete()to acquire the lock and open code the couple of callers that make additional checks under AIL lock.
- Drop the unnecessary ailp parameter from_delete().
- Drop the unused shutdown parameter from_remove()and open code the implementation.
Quoted 4 lines
In summary, this leaves a
_delete()variant that expects an AIL resident item and a_remove()helper that checks the AIL bit. Audit the existing callsites for use of the appropriate function and update as necessary.
Wouldn't it make sense to split this into separate patches for the items above? I have a bit of a hard time verifying this patch, even if the goal sounds just right.
Re: [PATCH v2 09/13] xfs: clean up AIL log item removal functions
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Brian Foster
Make the following changes to clean up both of these functions:
- Most callers of
xfs_trans_ail_delete()acquire the AIL lock just before the call. Update_delete()to acquire the lock and open code the couple of callers that make additional checks under AIL lock.
- Drop the unnecessary ailp parameter from_delete().
- Drop the unused shutdown parameter from_remove()and open code the implementation.
Quoted 9 lines
Quoted 4 lines
In summary, this leaves a
_delete()variant that expects an AIL resident item and a_remove()helper that checks the AIL bit. Audit the existing callsites for use of the appropriate function and update as necessary.Wouldn't it make sense to split this into separate patches for the items above? I have a bit of a hard time verifying this patch, even if the goal sounds just right.
This ended up as one patch pretty much because I couldn't figure out what the factoring steps needed to be until I came to the end result. I'll revisit splitting it up..
Brian
PATCH v2 10/13 xfs: combine xfs_trans_ail_[remove|delete]()
In reply to Brian FosterNow that the functions and callers of xfs_trans_ail_[remove|delete]() have been fixed up appropriately, the only difference between the two is the shutdown behavior. There are only a few callers of the _remove() variant, so make the shutdown conditional on the parameter and combine the two functions.
--- fs/xfs/xfs_dquot.c | 2 +- fs/xfs/xfs_dquot_item.c | 2 +- fs/xfs/xfs_inode_item.c | 2 +- fs/xfs/xfs_trans_ail.c | 27 ++++++++------------------- fs/xfs/xfs_trans_priv.h | 17 ----------------- 5 files changed, 11 insertions(+), 39 deletions(-)
fs/xfs/xfs_dquot.c +1 -1
@@ -1151,7 +1151,7 @@ xfs_qm_dqflush( out_abort: dqp->dq_flags &= ~XFS_DQ_DIRTY ; -xfs_trans_ail_remove (lip); +xfs_trans_ail_delete (lip,0 );xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); out_unlock:xfs_dqfunlock (dqp);
fs/xfs/xfs_dquot_item.c +1 -1
@@ -343,7 +343,7 @@ xfs_qm_qoff_logitem_relse(ASSERT (test_bit (XFS_LI_IN_AIL , &lip->li_flags) ||test_bit (XFS_LI_ABORTED , &lip->li_flags) ||XFS_FORCED_SHUTDOWN (lip->li_mountp)); -xfs_trans_ail_remove (lip); +xfs_trans_ail_delete (lip,0 );kmem_free (lip->li_lv_shadow);kmem_free (qoff); }
fs/xfs/xfs_inode_item.c +1 -1
@@ -763,7 +763,7 @@ xfs_iflush_abort(xfs_inode_log_item_t *iip = ip->i_itemp;if (iip) { -xfs_trans_ail_remove (&iip->ili_item); +xfs_trans_ail_delete (&iip->ili_item,0 ); iip->ili_logged =0 ;/* * Clear the ili_last_fields bits now that we know that the
fs/xfs/xfs_trans_ail.c +8 -19
@@ -850,25 +850,13 @@ xfs_ail_delete_one( }/** -* Remove a log items from the AIL +* Remove a log item from the AIL. * -* @xfs_trans_ail_delete_bulk takes an array of log items that all need to -* removed from the AIL. The caller is already holding the AIL lock, and done -* all the checks necessary to ensure the items passed in via @log_items are -* ready for deletion. This includes checking that the items are in the AIL. -* -* For each log item to be removed, unlink it from the AIL, clear the IN_AIL -* flag from the item and reset the item's lsn to 0. If we remove the first -* item in the AIL, update the log tail to match the new minimum LSN in the -* AIL. -* -* This function will not drop the AIL lock until all items are removed from -* the AIL to minimise the amount of lock traffic on the AIL. This does not -* greatly increase the AIL hold time, but does significantly reduce the amount -* of traffic on the lock, especially during IO completion. -* -* This function must be called with the AIL lock held. The lock is dropped -* before returning. +* For each log item to be removed, unlink it from the AIL, clear the IN_AIL +* flag from the item and reset the item's lsn to 0. If we remove the first item +* in the AIL, update the log tail to match the new minimum LSN in the AIL. If +* the item is not in the AIL, shut down if the caller has provided a shutdown +* type. Otherwise return quietly as this state is expected. */ void xfs_trans_ail_delete(
@@ -882,7 +870,7 @@ xfs_trans_ail_delete(spin_lock (&ailp->ail_lock);if (!test_bit (XFS_LI_IN_AIL , &lip->li_flags)) {spin_unlock (&ailp->ail_lock); -if (!XFS_FORCED_SHUTDOWN (mp)) { +if (shutdown_type && !XFS_FORCED_SHUTDOWN (mp)) {xfs_alert_tag (mp,XFS_PTAG_AILDELETE ,"%s: attempting to delete a log item that is not in the AIL" , __func__);
@@ -891,6 +879,7 @@ xfs_trans_ail_delete(return ; } +/* xfs_ail_update_finish() drops the AIL lock */ tail_lsn =xfs_ail_delete_one (ailp, lip);xfs_ail_update_finish (ailp, tail_lsn); }
fs/xfs/xfs_trans_priv.h +0 -17
@@ -96,23 +96,6 @@ void xfs_ail_update_finish(struct xfs_ail *ailp, xfs_lsn_t old_lsn)__releases (ailp->ail_lock);void xfs_trans_ail_delete (struct xfs_log_item *lip,int shutdown_type); -static inline void -xfs_trans_ail_remove ( -struct xfs_log_item *lip) -{ -struct xfs_ail *ailp = lip->li_ailp; -xfs_lsn_t tail_lsn; - -spin_lock (&ailp->ail_lock); -/* xfs_ail_update_finish() drops the AIL lock */ -if (test_bit (XFS_LI_IN_AIL , &lip->li_flags)) { - tail_lsn =xfs_ail_delete_one (ailp, lip); -xfs_ail_update_finish (ailp, tail_lsn); - }else { -spin_unlock (&ailp->ail_lock); - } -} -void xfs_ail_push (struct xfs_ail *,xfs_lsn_t );void xfs_ail_push_all (struct xfs_ail *);void xfs_ail_push_all_sync (struct xfs_ail *);
Signature · 1 line
2.21.1
Re: [PATCH v2 10/13] xfs: combine xfs_trans_ail_[remove|delete]()
Quoted 8 lines · Brian Foster
Now that the functions and callers of xfs_trans_ail_[remove|delete]() have been fixed up appropriately, the only difference between the two is the shutdown behavior. There are only a few callers of the
_remove()variant, so make the shutdown conditional on the parameter and combine the two functions.Suggested-by Dave Chinner <[email protected]>Signed-off-by Brian Foster <[email protected]>
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 11/13 xfs: remove unused iflush stale parameter
In reply to Brian FosterThe stale parameter was used to control the now unused shutdown parameter of xfs_trans_ail_remove().
--- fs/xfs/xfs_icache.c | 2 +- fs/xfs/xfs_inode.c | 2 +- fs/xfs/xfs_inode_item.c | 7 +++---- fs/xfs/xfs_inode_item.h | 2 +- 4 files changed, 6 insertions(+), 7 deletions(-)
fs/xfs/xfs_icache.c +1 -1
@@ -1128,7 +1128,7 @@ xfs_reclaim_inode(if (XFS_FORCED_SHUTDOWN (ip->i_mount)) {xfs_iunpin_wait (ip);/* xfs_iflush_abort() drops the flush lock */ -xfs_iflush_abort (ip, false); +xfs_iflush_abort (ip); goto reclaim; }if (xfs_ipincount (ip)) {
fs/xfs/xfs_inode.c +1 -1
@@ -3701,7 +3701,7 @@ xfs_iflush(return 0 ; abort: -xfs_iflush_abort (ip, false); +xfs_iflush_abort (ip); shutdown:xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );return error;
fs/xfs/xfs_inode_item.c +3 -4
@@ -757,10 +757,9 @@ xfs_iflush_done(*/ void xfs_iflush_abort ( -xfs_inode_t *ip, -bool stale) +struct xfs_inode *ip) { -xfs_inode_log_item_t *iip = ip->i_itemp; +struct xfs_inode_log_item *iip = ip->i_itemp;if (iip) {xfs_trans_ail_delete (&iip->ili_item,0 );
@@ -788,7 +787,7 @@ xfs_istale_done(struct xfs_buf *bp, struct xfs_log_item *lip) { - xfs_iflush_abort(INODE_ITEM(lip)->ili_inode, true); + xfs_iflush_abort(INODE_ITEM(lip)->ili_inode); }/*
fs/xfs/xfs_inode_item.h +1 -1
@@ -34,7 +34,7 @@ extern void xfs_inode_item_init(struct xfs_inode *, struct xfs_mount *);extern void xfs_inode_item_destroy (struct xfs_inode *);extern void xfs_iflush_done (struct xfs_buf *,struct xfs_log_item *);extern void xfs_istale_done (struct xfs_buf *,struct xfs_log_item *); -extern void xfs_iflush_abort (struct xfs_inode *,bool ); +extern void xfs_iflush_abort (struct xfs_inode *);extern int xfs_inode_item_format_convert (xfs_log_iovec_t *,struct xfs_inode_log_format *);
Signature · 1 line
2.21.1
Re: [PATCH v2 11/13] xfs: remove unused iflush stale parameter
Quoted 6 lines · Brian Foster
The stale parameter was used to control the now unused shutdown parameter of
xfs_trans_ail_remove().Signed-off-by Brian Foster <[email protected]>Reviewed-by Dave Chinner <[email protected]>Reviewed-by Allison Collins <[email protected]>
Looks good,
PATCH v2 12/13 xfs: random buffer write failure errortag
In reply to Brian FosterIntroduce an error tag to randomly fail async buffer writes. This is primarily to facilitate testing of the XFS error configuration mechanism.
--- fs/xfs/libxfs/xfs_errortag.h | 4 +++- fs/xfs/xfs_buf.c | 6 ++++++ fs/xfs/xfs_error.c | 3 +++ 3 files changed, 12 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_errortag.h +3 -1
@@ -55,7 +55,8 @@#define XFS_ERRTAG_FORCE_SCRUB_REPAIR 32#define XFS_ERRTAG_FORCE_SUMMARY_RECALC 33#define XFS_ERRTAG_IUNLINK_FALLBACK 34 -#define XFS_ERRTAG_MAX 35 +#define XFS_ERRTAG_BUF_IOERROR 35 +#define XFS_ERRTAG_MAX 36/* * Random factors for above tags, 1 means always, 2 means 1/2 time, etc.
@@ -95,5 +96,6 @@#define XFS_RANDOM_FORCE_SCRUB_REPAIR 1#define XFS_RANDOM_FORCE_SUMMARY_RECALC 1#define XFS_RANDOM_IUNLINK_FALLBACK (XFS_RANDOM_DEFAULT/10) +#define XFS_RANDOM_BUF_IOERROR XFS_RANDOM_DEFAULT#endif /* __XFS_ERRORTAG_H_ */
fs/xfs/xfs_buf.c +6 -0
@@ -1289,6 +1289,12 @@ xfs_buf_bio_end_io( struct bio *bio) {struct xfs_buf *bp = (struct xfs_buf *)bio->bi_private; +struct xfs_mount *mp = bp->b_mount; + +if (!bio->bi_status && + (bp->b_flags &XBF_WRITE ) && (bp->b_flags &XBF_ASYNC ) && +XFS_TEST_ERROR (false, mp,XFS_ERRTAG_BUF_IOERROR )) + bio->bi_status =errno_to_blk_status (-EIO );/* * don't overwrite existing errors - otherwise we can lose errors on
fs/xfs/xfs_error.c +3 -0
@@ -53,6 +53,7 @@ static unsigned int xfs_errortag_random_default[] = {XFS_RANDOM_FORCE_SCRUB_REPAIR ,XFS_RANDOM_FORCE_SUMMARY_RECALC ,XFS_RANDOM_IUNLINK_FALLBACK , + XFS_RANDOM_BUF_IOERROR, }; struct xfs_errortag_attr {
@@ -162,6 +163,7 @@ XFS_ERRORTAG_ATTR_RW(buf_lru_ref, XFS_ERRTAG_BUF_LRU_REF);XFS_ERRORTAG_ATTR_RW (force_repair,XFS_ERRTAG_FORCE_SCRUB_REPAIR );XFS_ERRORTAG_ATTR_RW (bad_summary,XFS_ERRTAG_FORCE_SUMMARY_RECALC );XFS_ERRORTAG_ATTR_RW (iunlink_fallback,XFS_ERRTAG_IUNLINK_FALLBACK ); +XFS_ERRORTAG_ATTR_RW (buf_ioerror,XFS_ERRTAG_BUF_IOERROR );static struct attribute *xfs_errortag_attrs[] = {XFS_ERRORTAG_ATTR_LIST (noerror),
@@ -199,6 +201,7 @@ static struct attribute *xfs_errortag_attrs[] = {XFS_ERRORTAG_ATTR_LIST (force_repair),XFS_ERRORTAG_ATTR_LIST (bad_summary),XFS_ERRORTAG_ATTR_LIST (iunlink_fallback), +XFS_ERRORTAG_ATTR_LIST (buf_ioerror), NULL, };
Signature · 1 line
2.21.1
Re: [PATCH v2 12/13] xfs: random buffer write failure errortag
Quoted 6 lines · Brian Foster
Introduce an error tag to randomly fail async buffer writes. This is primarily to facilitate testing of the XFS error configuration mechanism.
Signed-off-by Brian Foster <[email protected]>Reviewed-by Allison Collins <[email protected]>
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 12/13] xfs: random buffer write failure errortag
In reply to Brian Fosterfs/xfs/xfs_buf.c xfs_buf_bio_end_io( · line 1297
Quoted 10 lines · Brian Foster
fs/xfs/xfs_buf.c +6
-0@@ -1289,6 +1289,12 @@ xfs_buf_bio_end_io( struct bio *bio) {struct xfs_buf *bp = (struct xfs_buf *)bio->bi_private; +struct xfs_mount *mp = bp->b_mount; + +if (!bio->bi_status && + (bp->b_flags &XBF_WRITE ) && (bp->b_flags &XBF_ASYNC ) && +XFS_TEST_ERROR (false, mp,XFS_ERRTAG_BUF_IOERROR )) + bio->bi_status =errno_to_blk_status (-EIO );
Please just use BLK_STS_IOERR directly here.
Re: [PATCH v2 12/13] xfs: random buffer write failure errortag
Quoted 14 lines · Christoph Hellwig
fs/xfs/xfs_buf.c xfs_buf_bio_end_io( · line 1297
Quoted 10 lines · Brian Foster
fs/xfs/xfs_buf.c +6
-0@@ -1289,6 +1289,12 @@ xfs_buf_bio_end_io( struct bio *bio) {struct xfs_buf *bp = (struct xfs_buf *)bio->bi_private; +struct xfs_mount *mp = bp->b_mount; + +if (!bio->bi_status && + (bp->b_flags &XBF_WRITE ) && (bp->b_flags &XBF_ASYNC ) && +XFS_TEST_ERROR (false, mp,XFS_ERRTAG_BUF_IOERROR )) + bio->bi_status =errno_to_blk_status (-EIO );Please just use
BLK_STS_IOERRdirectly here.
Ok, fixed.
Brian
PATCH v2 13/13 xfs: remove unused shutdown types
In reply to Brian FosterBoth types control shutdown messaging and neither is used in the current codebase.
--- fs/xfs/xfs_fsops.c | 5 +---- fs/xfs/xfs_mount.h | 2 -- 2 files changed, 1 insertion(+), 6 deletions(-)
fs/xfs/xfs_fsops.c +1 -4
@@ -504,10 +504,7 @@ xfs_do_force_shutdown( }else if (logerror) {xfs_alert_tag (mp,XFS_PTAG_SHUTDOWN_LOGERROR ,"Log I/O Error Detected. Shutting down filesystem" ); - }else if (flags &SHUTDOWN_DEVICE_REQ ) { -xfs_alert_tag (mp,XFS_PTAG_SHUTDOWN_IOERROR , -"All device paths lost. Shutting down filesystem" ); - }else if (!(flags &SHUTDOWN_REMOTE_REQ )) { + }else {xfs_alert_tag (mp,XFS_PTAG_SHUTDOWN_IOERROR ,"I/O Error Detected. Shutting down filesystem" ); }
fs/xfs/xfs_mount.h +0 -2
@@ -259,8 +259,6 @@ void xfs_do_force_shutdown(struct xfs_mount *mp, int flags, char *fname,#define SHUTDOWN_LOG_IO_ERROR 0x0002/* write attempt to the log failed */ #define SHUTDOWN_FORCE_UMOUNT 0x0004/* shutdown from a forced unmount */ #define SHUTDOWN_CORRUPT_INCORE 0x0008/* corrupt in-memory data structures */ -#define SHUTDOWN_REMOTE_REQ 0x0010/* shutdown came from remote cell */ -#define SHUTDOWN_DEVICE_REQ 0x0020/* failed all paths to the device */ /* * Flags for xfs_mountfs
Signature · 1 line
2.21.1
Re: [PATCH v2 13/13] xfs: remove unused shutdown types
Quoted 4 lines · Brian Foster
Both types control shutdown messaging and neither is used in the current codebase.
Signed-off-by Brian Foster <[email protected]>
Yup, historical artifacts, stuff never implemented on Linux.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v2 13/13] xfs: remove unused shutdown types
In reply to Brian FosterQuoted 2 lines · Brian Foster
Both types control shutdown messaging and neither is used in the current codebase.
Hmm, I'm pretty sure I submitted the same patch a few weeks ago :)
Because of that it obviously looks fine, of course: