xfs: a couple AIL pushing trylock fixes
cover xfs: a couple AIL pushing trylock fixes
Hi all,
Here's a couple more small fixes that fell out of the auto relog work. The dquot issue is actually a deadlock vector if we randomly relog dquot buffers (which is only done for test purposes), but I figure we should handle dquot buffers similar to how inode buffers are handled. Thoughts, reviews, flames appreciated.
Brian
Brian Foster (2):
xfs: trylock underlying buffer on dquot flush
xfs: return locked status of inode buffer on xfsaild push
fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_inode_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 4 files changed, 16 insertions(+), 10 deletions(-)
Signature · 1 line
2.21.1
PATCH 2/2 xfs: return locked status of inode buffer on xfsaild push
If the inode buffer backing a particular inode is locked, xfs_iflush() returns -EAGAIN and xfs_inode_item_push() skips the inode. It still returns success to xfsaild, however, which bypasses the xfsaild backoff heuristic. Update xfs_inode_item_push() to return locked status if the inode buffer couldn't be locked.
--- fs/xfs/xfs_inode_item.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/xfs/xfs_inode_item.c +2 -1
@@ -552,7 +552,8 @@ xfs_inode_item_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:
Signature · 1 line
2.21.1
PATCH 1/2 xfs: trylock underlying buffer on dquot flush
In reply to Brian FosterA dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update xfs_qm_dqflush() to trylock the buffer, similar to how inode buffers are handled, and return -EAGAIN if the lock fails. Fix up any callers that don't currently handle the error properly.
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)
fs/xfs/xfs_dquot.c +3 -3
@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;
@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/*
fs/xfs/xfs_dquot_item.c +2 -1
@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:
fs/xfs/xfs_qm.c +9 -5
@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;
@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need
@@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }
@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/*
Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
Quoted 6 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.
Looks good, seems like the two remaining xfs_qm_dqflush have sensible -EAGAIN handling.
Re: [PATCH 2/2] xfs: return locked status of inode buffer on xfsaild push
In reply to Brian FosterQuoted 7 lines · Brian Foster
If the inode buffer backing a particular inode is locked,
xfs_iflush()returns-EAGAINandxfs_inode_item_push()skips the inode. It still returns success to xfsaild, however, which bypasses the xfsaild backoff heuristic. Updatexfs_inode_item_push()to return locked status if the inode buffer couldn't be locked.Signed-off-by Brian Foster <[email protected]>
Looks good,
Quoted 7 lines · Brian Foster
Hi all,
Here's a couple more small fixes that fell out of the auto relog work. The dquot issue is actually a deadlock vector if we randomly relog dquot buffers (which is only done for test purposes), but I figure we should handle dquot buffers similar to how inode buffers are handled. Thoughts, reviews, flames appreciated.
Oops, I missed this one, will review now...
Do you think there needs to be an explicit testcase for this? Or are the current generic/{388,475} good enough? I'm pretty sure I've seen this exact deadlock on them every now and again, so we're probably covered.
--D
Quoted 15 lines
Brian
Brian Foster (2):
xfs: trylock underlying buffer on dquot flush
xfs: return locked status of inode buffer on xfsaild pushfs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_inode_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 4 files changed, 16 insertions(+), 10 deletions(-)Signature · 1 line
2.21.1
Re: [PATCH 2/2] xfs: return locked status of inode buffer on xfsaild push
In reply to Brian FosterQuoted 7 lines · Brian Foster
If the inode buffer backing a particular inode is locked,
xfs_iflush()returns-EAGAINandxfs_inode_item_push()skips the inode. It still returns success to xfsaild, however, which bypasses the xfsaild backoff heuristic. Updatexfs_inode_item_push()to return locked status if the inode buffer couldn't be locked.Signed-off-by Brian Foster <[email protected]>
Seems pretty straightforward,
--D
Quoted 21 lines
--- fs/xfs/xfs_inode_item.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)fs/xfs/xfs_inode_item.c +2
-1@@ -552,7 +552,8 @@ xfs_inode_item_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Brian FosterQuoted 8 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>
Is xfs_qm_dquot_isolate returning LRU_RETRY an acceptable resolution (as opposed to, say, LRU_SKIP) for xfs_qm_dqflush returning -EAGAIN?
--D
Quoted 95 lines
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:fs/xfs/xfs_qm.c +9
-5@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need @@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/* Signature · 1 line
2.21.1
Quoted 9 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
Hi all,
Here's a couple more small fixes that fell out of the auto relog work. The dquot issue is actually a deadlock vector if we randomly relog dquot buffers (which is only done for test purposes), but I figure we should handle dquot buffers similar to how inode buffers are handled. Thoughts, reviews, flames appreciated.
Quoted 7 lines
Oops, I missed this one, will review now...
Do you think there needs to be an explicit testcase for this? Or are the current generic/{388,475} good enough? I'm pretty sure I've seen this exact deadlock on them every now and again, so we're probably covered.
I'm actually not aware of a related upstream deadlock. That doesn't mean there isn't one of course, but the problem I hit was related to the random buffer relogging stuff in the auto relog series. I split these out because xfsaild is intended to be mostly async, so they seemed like a generic fixups..
Brian
Quoted 19 lines
--D
Quoted 15 lines
Brian
Brian Foster (2):
xfs: trylock underlying buffer on dquot flush
xfs: return locked status of inode buffer on xfsaild pushfs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_inode_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 4 files changed, 16 insertions(+), 10 deletions(-)Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Darrick J. WongQuoted 8 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.
Quoted 5 lines
Quoted 1 line
Signed-off-by Brian Foster <[email protected]>Is
xfs_qm_dquot_isolatereturningLRU_RETRYan acceptable resolution (as opposed to, say,LRU_SKIP) forxfs_qm_dqflushreturning-EAGAIN?
Hmm.. this is reclaim so I suppose LRU_SKIP would be more appropriate than retry (along with more consistent with the other trylock failures in that function). Ok with something like the following?
@@ -461,7 +461,11 @@ xfs_qm_dquot_isolate(
spin_unlock(lru_lock);
error = xfs_qm_dqflush(dqp, &bp);
- if (error)
+ if (error == -EAGAIN) {
+ xfs_dqunlock(dqp);
+ spin_lock(lru_lock);
+ goto out_miss_busy;
+ } else if (error)
goto out_unlock_dirty;
xfs_buf_delwri_queue(bp, &isol->buffers);
Brian
Quoted 98 lines
--D
Quoted 95 lines
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:fs/xfs/xfs_qm.c +9
-5@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need @@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/* Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
Quoted 20 lines · Brian Foster
Quoted 8 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Quoted 5 lines
Quoted 1 line
Signed-off-by Brian Foster <[email protected]>Is
xfs_qm_dquot_isolatereturningLRU_RETRYan acceptable resolution (as opposed to, say,LRU_SKIP) forxfs_qm_dqflushreturning-EAGAIN?Hmm.. this is reclaim so I suppose
LRU_SKIPwould be more appropriate than retry (along with more consistent with the other trylock failures in that function). Ok with something like the following?
Review xfs_qm_dquot_isolate( · line 469
Quoted 10 lines
(quoted diff) +5
-1@@ -461,7 +461,11 @@ xfs_qm_dquot_isolate( spin_unlock(lru_lock); error = xfs_qm_dqflush(dqp, &bp); - if (error) + if (error == -EAGAIN) { + xfs_dqunlock(dqp); + spin_lock(lru_lock); + goto out_miss_busy; + } else if (error) goto out_unlock_dirty;
Then again, is it safe to skip from here once we've cycled the lru_lock?
Brian
Quoted 104 lines
(quoted diff) +0
-0xfs_buf_delwri_queue(bp, &isol->buffers);Brian
Quoted 98 lines
--D
Quoted 95 lines
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:fs/xfs/xfs_qm.c +9
-5@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need @@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/* Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Brian FosterQuoted 22 lines · Brian Foster
Quoted 8 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Quoted 5 lines
Quoted 1 line
Signed-off-by Brian Foster <[email protected]>Is
xfs_qm_dquot_isolatereturningLRU_RETRYan acceptable resolution (as opposed to, say,LRU_SKIP) forxfs_qm_dqflushreturning-EAGAIN?Hmm.. this is reclaim so I suppose
LRU_SKIPwould be more appropriate than retry (along with more consistent with the other trylock failures in that function). Ok with something like the following?(quoted diff) +0
-0@@ -461,7 +461,11 @@ xfs_qm_dquot_isolate( spin_unlock(lru_lock);
Review xfs_qm_dquot_isolate( · line 471
Quoted 10 lines
(quoted diff) +5
-1error = xfs_qm_dqflush(dqp, &bp); - if (error) + if (error == -EAGAIN) { + xfs_dqunlock(dqp); + spin_lock(lru_lock); + goto out_miss_busy; + } else if (error) goto out_unlock_dirty; xfs_buf_delwri_queue(bp, &isol->buffers);
Yeah, looks good to me.
--D
Quoted 102 lines
Brian
Quoted 98 lines
--D
Quoted 95 lines
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:fs/xfs/xfs_qm.c +9
-5@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need @@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/* Signature · 1 line
2.21.1
Darrick J. Wong
Reviewed-byRe: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Brian FosterQuoted 23 lines · Brian Foster
Quoted 22 lines · Brian Foster
Quoted 8 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Quoted 5 lines
Quoted 1 line
Signed-off-by Brian Foster <[email protected]>Is
xfs_qm_dquot_isolatereturningLRU_RETRYan acceptable resolution (as opposed to, say,LRU_SKIP) forxfs_qm_dqflushreturning-EAGAIN?Hmm.. this is reclaim so I suppose
LRU_SKIPwould be more appropriate than retry (along with more consistent with the other trylock failures in that function). Ok with something like the following?(quoted diff) +0
-0@@ -461,7 +461,11 @@ xfs_qm_dquot_isolate( spin_unlock(lru_lock);
Quoted 10 lines
Review xfs_qm_dquot_isolate( · line 469
Quoted 8 lines
(quoted diff) +5
-1error = xfs_qm_dqflush(dqp, &bp); - if (error) + if (error == -EAGAIN) { + xfs_dqunlock(dqp); + spin_lock(lru_lock); + goto out_miss_busy; + } else if (error) goto out_unlock_dirty;Then again, is it safe to skip from here once we've cycled the
lru_lock?
DOH. Yeah, I missed that we cycled the lru lock and therefore have to LRU_RETRY. So I guess the original patch was fine:
--D
Quoted 107 lines
Brian
Quoted 104 lines
(quoted diff) +0
-0xfs_buf_delwri_queue(bp, &isol->buffers);Brian
Quoted 98 lines
--D
Quoted 95 lines
--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;spin_lock (&lip->li_ailp->ail_lock); out_unlock:fs/xfs/xfs_qm.c +9
-5@@ -121,12 +121,11 @@ xfs_qm_dqpurge( {struct xfs_mount *mp = dqp->q_mount;struct xfs_quotainfo *qi = mp->m_quotainfo; +int error = -EAGAIN ;xfs_dqlock (dqp); -if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) { -xfs_dqunlock (dqp); -return -EAGAIN ; - } +if ((dqp->dq_flags &XFS_DQ_FREEING ) || dqp->q_nrefs !=0 ) + goto out_unlock; dqp->dq_flags |=XFS_DQ_FREEING ;@@ -139,7 +138,6 @@ xfs_qm_dqpurge(*/ if (XFS_DQ_IS_DIRTY (dqp)) {struct xfs_buf *bp =NULL ; -int error;/* * We don't care about getting disk errors here. We need @@ -149,6 +147,8 @@ xfs_qm_dqpurge(if (!error) { error =xfs_bwrite (bp);xfs_buf_relse (bp); + }else if (error == -EAGAIN ) { + goto out_unlock; }xfs_dqflock (dqp); }@@ -174,6 +174,10 @@ xfs_qm_dqpurge(xfs_qm_dqdestroy (dqp);return 0 ; + +out_unlock: +xfs_dqunlock (dqp); +return error; }/* Signature · 1 line
2.21.1
Quoted 18 lines · Brian Foster
Quoted 9 lines · Darrick J. Wong
Quoted 7 lines · Brian Foster
Hi all,
Here's a couple more small fixes that fell out of the auto relog work. The dquot issue is actually a deadlock vector if we randomly relog dquot buffers (which is only done for test purposes), but I figure we should handle dquot buffers similar to how inode buffers are handled. Thoughts, reviews, flames appreciated.
Quoted 7 lines
Oops, I missed this one, will review now...
Do you think there needs to be an explicit testcase for this? Or are the current generic/{388,475} good enough? I'm pretty sure I've seen this exact deadlock on them every now and again, so we're probably covered.
Quoted 5 lines
I'm actually not aware of a related upstream deadlock. That doesn't mean there isn't one of course, but the problem I hit was related to the random buffer relogging stuff in the auto relog series. I split these out because xfsaild is intended to be mostly async, so they seemed like a generic fixups..
<nod> FWIW I'd traced a generic/475 shutdown hang as far as "the AIL seems to be stuck on a locked dquot buffer" but haven't really had a chance to look into what was going on at the time.
Whereas before it would usually hang if I let it run more than about 15 minutes, now I've been able to get it to run all night to completion.
--D
Quoted 22 lines
Brian
Quoted 19 lines
--D
Quoted 15 lines
Brian
Brian Foster (2):
xfs: trylock underlying buffer on dquot flush
xfs: return locked status of inode buffer on xfsaild pushfs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_inode_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 4 files changed, 16 insertions(+), 10 deletions(-)Signature · 1 line
2.21.1
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Brian FosterQuoted 43 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +0
-0
fs/xfs/xfs_dquot_item.c xfs_qm_dquot_logitem_push( · line 193
Quoted 6 lines
fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;
Doesn't xfs_inode_item_push() also have this problem in that it doesn't handle -EAGAIN properly?
Also, we can get -EIO, -EFSCORRUPTED, etc here. They probably shouldn't return XFS_ITEM_SUCCESS, either....
Otherwise seems OK.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
Quoted 38 lines · Dave Chinner
Quoted 37 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }
Quoted 15 lines
fs/xfs/xfs_dquot_item.c xfs_qm_dquot_logitem_push( · line 193
Quoted 12 lines
fs/xfs/xfs_dquot.c +0
-0/* fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;Doesn't
xfs_inode_item_push()also have this problem in that it doesn't handle-EAGAINproperly?
... and now I see this is the second patch in the series...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
In reply to Dave ChinnerQuoted 54 lines · Dave Chinner
Quoted 43 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +0
-0fs/xfs/xfs_dquot_item.c xfs_qm_dquot_logitem_push( · line 193
Quoted 6 lines
fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;Doesn't
xfs_inode_item_push()also have this problem in that it doesn't handle-EAGAINproperly?
Quoted 3 lines
Also, we can get
-EIO,-EFSCORRUPTED, etc here. They probably shouldn't returnXFS_ITEM_SUCCESS, either....
Good point. I'm actually not sure what we should return in that case given the item return codes all seem to assume a valid state. We could define an XFS_ITEM_ERROR return, but I'm not sure it's worth it for what is currently stat/tracepoint logic in the caller. Perhaps a broader rework of error handling in this context is in order that would lift generic (fatal) error handling into xfsaild. E.g., I see that xfs_qm_dqflush() is inconsistent by itself in that the item is removed from the AIL if we're already shut down, but not if that function invokes the shutdown; we shutdown if the direct xfs_dqblk_verify() call fails but not if the read verifier (which also looks like it calls xfs_dqblk_verify() on every on-disk dquot) returns -EFSCORRUPTED, etc. It might make some sense to let iop_push() return negative error codes if that facilitates consistent error handling...
Brian
Quoted 9 lines
Otherwise seems OK.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
Quoted 59 lines · Brian Foster
Quoted 54 lines · Dave Chinner
Quoted 43 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +0
-0fs/xfs/xfs_dquot_item.c xfs_qm_dquot_logitem_push( · line 193
Quoted 6 lines
fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;Doesn't
xfs_inode_item_push()also have this problem in that it doesn't handle-EAGAINproperly?Quoted 3 lines
Also, we can get
-EIO,-EFSCORRUPTED, etc here. They probably shouldn't returnXFS_ITEM_SUCCESS, either....
Quoted 6 lines
Good point. I'm actually not sure what we should return in that case given the item return codes all seem to assume a valid state. We could define an
XFS_ITEM_ERRORreturn, but I'm not sure it's worth it for what is currently stat/tracepoint logic in the caller. Perhaps a broader rework of error handling in this context is in order that would lift generic (fatal) error handling into xfsaild.
Yeah, that's where my thoughts were heading as well.
Quoted 8 lines
E.g., I see that
xfs_qm_dqflush()is inconsistent by itself in that the item is removed from the AIL if we're already shut down, but not if that function invokes the shutdown; we shutdown if the directxfs_dqblk_verify()call fails but not if the read verifier (which also looks like it callsxfs_dqblk_verify()on every on-disk dquot) returns-EFSCORRUPTED, etc. It might make some sense to letiop_push()return negative error codes if that facilitates consistent error handling...
Yes, it's a bit of a mess. I suspect that what we should be doing here is pulling the failed buffer write retry code up into the main push loop. That is, we can set LI_FAILED on log items that fail to flush, either directly at submit time, or at IO completion for write errors.
Then we can have the main AIL loop set LI_FAILED on push failures, and also the main loop detect LI_FAILED directly and call a new ->iop_resubmit() function rather than having to handle that the resubmit cases as special cases in every ->iop_push() path.
That seems like a much cleaner way of handling submission failure and retries for all log item types that need it compared to the way we currently handle it for buffers...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 1/2] xfs: trylock underlying buffer on dquot flush
Quoted 84 lines · Dave Chinner
Quoted 59 lines · Brian Foster
Quoted 54 lines · Dave Chinner
Quoted 43 lines · Brian Foster
A dquot flush currently blocks on the buffer lock for the underlying dquot buffer. In turn, this causes xfsaild to block rather than continue processing other items in the meantime. Update
xfs_qm_dqflush()to trylock the buffer, similar to how inode buffers are handled, and return-EAGAINif the lock fails. Fix up any callers that don't currently handle the error properly.Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_dquot_item.c | 3 ++- fs/xfs/xfs_qm.c | 14 +++++++++----- 3 files changed, 14 insertions(+), 9 deletions(-)fs/xfs/xfs_dquot.c +3
-3@@ -1105,8 +1105,8 @@ xfs_qm_dqflush(* Get the buffer containing the on-disk dquot */ error =xfs_trans_read_buf (mp,NULL , mp->m_ddev_targp, dqp->q_blkno, - mp->m_quotainfo->qi_dqchunklen,0 , &bp, - &xfs_dquot_buf_ops); + mp->m_quotainfo->qi_dqchunklen,XBF_TRYLOCK , + &bp, &xfs_dquot_buf_ops);if (error) goto out_unlock;@@ -1177,7 +1177,7 @@ xfs_qm_dqflush( out_unlock:xfs_dqfunlock (dqp); -return -EIO ; +return error; }/* fs/xfs/xfs_dquot_item.c +0
-0fs/xfs/xfs_dquot_item.c xfs_qm_dquot_logitem_push( · line 193
Quoted 6 lines
fs/xfs/xfs_dquot_item.c +2
-1@@ -189,7 +189,8 @@ xfs_qm_dquot_logitem_push(if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - } + }else if (error == -EAGAIN ) + rval =XFS_ITEM_LOCKED ;Doesn't
xfs_inode_item_push()also have this problem in that it doesn't handle-EAGAINproperly?Quoted 3 lines
Also, we can get
-EIO,-EFSCORRUPTED, etc here. They probably shouldn't returnXFS_ITEM_SUCCESS, either....Quoted 6 lines
Good point. I'm actually not sure what we should return in that case given the item return codes all seem to assume a valid state. We could define an
XFS_ITEM_ERRORreturn, but I'm not sure it's worth it for what is currently stat/tracepoint logic in the caller. Perhaps a broader rework of error handling in this context is in order that would lift generic (fatal) error handling into xfsaild.Yeah, that's where my thoughts were heading as well.
Quoted 8 lines
E.g., I see that
xfs_qm_dqflush()is inconsistent by itself in that the item is removed from the AIL if we're already shut down, but not if that function invokes the shutdown; we shutdown if the directxfs_dqblk_verify()call fails but not if the read verifier (which also looks like it callsxfs_dqblk_verify()on every on-disk dquot) returns-EFSCORRUPTED, etc. It might make some sense to letiop_push()return negative error codes if that facilitates consistent error handling...Yes, it's a bit of a mess. I suspect that what we should be doing here is pulling the failed buffer write retry code up into the main push loop. That is, we can set
LI_FAILEDon log items that fail to flush, either directly at submit time, or at IO completion for write errors.
Quoted 5 lines
Then we can have the main AIL loop set
LI_FAILEDon push failures, and also the main loop detectLI_FAILEDdirectly and call a new ->iop_resubmit() function rather than having to handle that the resubmit cases as special cases in every ->iop_push() path.
I'm not sure we want to use LI_FAILED in failure to flush (i.e. push failure) situations because it's currently used specifically to indicate that a particular item requires resubmit when it already has been successfully flushed. This avoids the need for a post I/O error push to retry an already locked flush lock (and flush attempt) and subsequently cause the item to remain stuck on the AIL. It still might make sense to refactor the existing LI_FAILED implementation into ->iop_resubmit() callbacks for those items that use it, though.
That also doesn't preclude refactoring some sort of generic push failure error handling into xfsaild for the sake of consistency. It's just not immediately clear to me what it should look like. Perhaps I'll poke at it a bit once I get the next rfc of the relog work settled and posted (soon)..
Brian
Quoted 11 lines
That seems like a much cleaner way of handling submission failure and retries for all log item types that need it compared to the way we currently handle it for buffers...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]