Fix use-after-free with log and quotas
PATCH 1/1 Fix use-after-free with log and quotas
Destroying the quota stuff on unmount can access the log - ie XFS_QM_DONE() ends up in xfs_dqunlock() which calls xfs_trans_unlocked_item() and then xfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier in xfs_unmountfs() before the call to xfs_log_unmount(). Moving XFS_QM_DONE() up near XFS_QM_DQPURGEALL() seems like a good spot.
fs/xfs/xfs_mount.c +0 -0
@@ -1245,6 +1245,9 @@ xfs_unmountfs(
XFS_QM_DQPURGEALL(mp, XFS_QMOPT_QUOTALL | XFS_QMOPT_UMOUNTING);
+ if (mp->m_quotainfo)
+ XFS_QM_DONE(mp);
+
/*
* Flush out the log synchronously so that we know for sure
* that nothing is pinned. This is important because bflush()
@@ -1297,8 +1300,6 @@ xfs_unmountfs(
xfs_errortag_clearall(mp, 0); #endif xfs_free_perag(mp);
- if (mp->m_quotainfo)
- XFS_QM_DONE(mp);
}
STATIC void
Quoted 6 lines · Lachlan McIlroy
Destroying the quota stuff on unmount can access the log - ie
XFS_QM_DONE()ends up inxfs_dqunlock()which callsxfs_trans_unlocked_item()and thenxfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier inxfs_unmountfs()before the call toxfs_log_unmount(). MovingXFS_QM_DONE()up nearXFS_QM_DQPURGEALL()seems like a good spot.
Yeah, I have something like this in my large mount/unmount audit.
Looks good, and I have to rebase all the other changes anyway.
Quoted 6 lines · Lachlan McIlroy
Destroying the quota stuff on unmount can access the log - ie
XFS_QM_DONE()ends up inxfs_dqunlock()which callsxfs_trans_unlocked_item()and thenxfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier inxfs_unmountfs()before the call toxfs_log_unmount(). MovingXFS_QM_DONE()up nearXFS_QM_DQPURGEALL()seems like a good spot.
FWIW, has this been actually seen in the real world?
xfs_trans_unlocked_item() only does stuff if the log item is in the AIL. If we've already destroyed the log, then we should have already torn down the AIL and there should be no log items in the system that are in the AIL....
What am I missing here?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 9 lines · Dave Chinner
Quoted 6 lines · Lachlan McIlroy
Destroying the quota stuff on unmount can access the log - ie
XFS_QM_DONE()ends up inxfs_dqunlock()which callsxfs_trans_unlocked_item()and thenxfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier inxfs_unmountfs()before the call toxfs_log_unmount(). MovingXFS_QM_DONE()up nearXFS_QM_DQPURGEALL()seems like a good spot.FWIW, has this been actually seen in the real world?
Yes. And easy to reproduce too.
Quoted 5 lines
xfs_trans_unlocked_item()only does stuff if the log item is in the AIL. If we've already destroyed the log, then we should have already torn down the AIL and there should be no log items in the system that are in the AIL....
That should be the case but clearly not happening. Pete is investigating an issue right now where a dquot is not getting removed from the AIL when it should. Until we've got to the bottom of that problem I'd prefer to at least avoid this use after free issue.
Quoted 11 lines · Lachlan McIlroy
Quoted 9 lines · Dave Chinner
Quoted 6 lines · Lachlan McIlroy
Destroying the quota stuff on unmount can access the log - ie
XFS_QM_DONE()ends up inxfs_dqunlock()which callsxfs_trans_unlocked_item()and thenxfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier inxfs_unmountfs()before the call toxfs_log_unmount(). MovingXFS_QM_DONE()up nearXFS_QM_DQPURGEALL()seems like a good spot.FWIW, has this been actually seen in the real world?
Yes. And easy to reproduce too.
Care to provide details about the test case, then? I can't help if you keep me in the dark....
Quoted 6 lines
Quoted 2 lines
torn down the AIL and there should be no log items in the system that are in the AIL....
That should be the case but clearly not happening. Pete is investigating an issue right now where a dquot is not getting removed from the AIL when it should. Until we've got to the bottom of that problem I'd prefer to at least avoid this use after free issue.
No point in putting a bandaid in if you're already in the process of trying to find the real cause....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Lachlan McIlroy
Quoted 7 lines · Dave Chinner
Quoted 6 lines · Lachlan McIlroy
Destroying the quota stuff on unmount can access the log - ie
XFS_QM_DONE()ends up inxfs_dqunlock()which callsxfs_trans_unlocked_item()and thenxfs_log_move_tail(). By this time the log has already been destroyed. Just move the cleanup of the quota code earlier inxfs_unmountfs()before the call toxfs_log_unmount(). MovingXFS_QM_DONE()up nearXFS_QM_DQPURGEALL()seems like a good spot.
Quoted 5 lines
Quoted 2 lines
Quoted 1 line
FWIW, has this been actually seen in the real world?
Yes. And easy to reproduce too.
Care to provide details about the test case, then? I can't help if you keep me in the dark....
XFSQA test 083 hits this almost every run when quotas are enabled.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
torn down the AIL and there should be no log items in the system that are in the AIL....
That should be the case but clearly not happening. Pete is investigating an issue right now where a dquot is not getting removed from the AIL when it should. Until we've got to the bottom of that problem I'd prefer to at least avoid this use after free issue.
No point in putting a bandaid in if you're already in the process of trying to find the real cause....
It's not a band-aid - it's a perfectly valid change to make. We don't know if this other problem is related nor do we know if it will fix this use-after-free either. There's no reason to let other people hit this panic if we can avoid it.