xfs: copy li_lsn before dropping AIL lock
PATCH 1/1 xfs: copy li_lsn before dropping AIL lock
Access to log items on the AIL is generally protected by m_ail_lock; this is particularly needed when we're getting or setting the 64-bit li_lsn on a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.
This can result in a partially-zeroed log->l_tail_lsn if xfs_trans_ail_delete is racing with xfs_trans_ail_update, and in at least some cases, this can leave the l_tail_lsn with a zero cycle number, which means xlog_space_left will think the log is full (unless CONFIG_XFS_DEBUG is set, in which case we'll trip an ASSERT), leading to processes stuck forever in xlog_grant_log_space.
Thanks to Adrian VanderSpek for first spotting the race potential and to Dave Chinner for debug assistance.
--- fs/xfs/xfs_trans_ail.c | 23 ++++++++++++++++++++--- 1 files changed, 20 insertions(+), 3 deletions(-)
fs/xfs/xfs_trans_ail.c +20 -3
@@ -467,6 +467,7 @@ xfs_trans_ail_update( {xfs_log_item_t *dlip =NULL ;xfs_log_item_t *mlip;/* ptr to minimum lip */ +xfs_lsn_t tail_lsn; mlip =xfs_ail_min (ailp);
@@ -483,8 +484,16 @@ xfs_trans_ail_update(if (mlip == dlip) { mlip =xfs_ail_min (ailp); +/* +* It is not safe to access mlip after the AIL lock is +* dropped, so we must get a copy of li_lsn before we do +* so. This is especially important on 32-bit platforms +* where accessing and updating 64-bit values like li_lsn +* is not atomic. +*/ + tail_lsn = mlip->li_lsn;spin_unlock (&ailp->xa_lock); -xfs_log_move_tail (ailp->xa_mount, mlip->li_lsn); +xfs_log_move_tail (ailp->xa_mount, tail_lsn); }else {spin_unlock (&ailp->xa_lock); }
@@ -514,6 +523,7 @@ xfs_trans_ail_delete( {xfs_log_item_t *dlip;xfs_log_item_t *mlip; +xfs_lsn_t tail_lsn;if (lip->li_flags &XFS_LI_IN_AIL ) { mlip =xfs_ail_min (ailp);
@@ -527,9 +537,16 @@ xfs_trans_ail_delete(if (mlip == dlip) { mlip =xfs_ail_min (ailp); +/* +* It is not safe to access mlip after the AIL lock +* is dropped, so we must get a copy of li_lsn +* before we do so. This is especially important +* on 32-bit platforms where accessing and updating +* 64-bit values like li_lsn is not atomic. +*/ + tail_lsn = mlip ? mlip->li_lsn :0 ;spin_unlock (&ailp->xa_lock); -xfs_log_move_tail (ailp->xa_mount, - (mlip ? mlip->li_lsn :0 )); +xfs_log_move_tail (ailp->xa_mount, tail_lsn); }else {spin_unlock (&ailp->xa_lock); }
Signature · 7 lines
Nathaniel W. Turner http://houseofnate.net/ _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Quoted 6 lines · Nathaniel W. Turner
Access to log items on the AIL is generally protected by
m_ail_lock; this is particularly needed when we're getting or setting the 64-bitli_lsnon a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.
Quoted 6 lines
This can result in a partially-zeroed
log->l_tail_lsnifxfs_trans_ail_deleteis racing withxfs_trans_ail_update, and in at least some cases, this can leave thel_tail_lsnwith a zero cycle number, which meansxlog_space_leftwill think the log is full (unlessCONFIG_XFS_DEBUGis set, in which case we'll trip an ASSERT), leading to processes stuck forever inxlog_grant_log_space.
Might this also cause this oops?
http://www.kerneloops.org/raw.php?rawid=944396&msgid=
It's been shoving up a few times recently.
Anyway, the patch looks good to me, but I wonder if we should abstract the li_lsn handling a bit more to avoid easily running into this kind of problems.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Quoted 14 lines · Christoph Hellwig
Quoted 6 lines · Nathaniel W. Turner
Access to log items on the AIL is generally protected by
m_ail_lock; this is particularly needed when we're getting or setting the 64-bitli_lsnon a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.Quoted 6 lines
This can result in a partially-zeroed
log->l_tail_lsnifxfs_trans_ail_deleteis racing withxfs_trans_ail_update, and in at least some cases, this can leave thel_tail_lsnwith a zero cycle number, which meansxlog_space_leftwill think the log is full (unlessCONFIG_XFS_DEBUGis set, in which case we'll trip an ASSERT), leading to processes stuck forever inxlog_grant_log_space.
Quoted 5 lines
Might this also cause this oops?
http://www.kerneloops.org/raw.php?rawid=944396&msgid=
It's been shoving up a few times recently.
I don't think so. That trace is in xfs_close_devices(), which is called after xfs_unmountfs->xfs_log_unmount->xlog_dealloc, which tends to imply that mp->m_log is NULL.
i.e. we have IO being flushed when no IO should be pending. Given that there is a xfs->bdstrat_cb->xfs_bioerror trace in there, it looks like the buffer could not be written because the system is in a forced shutdown situation. I have no idea what that buffer might be or why it wasn't issued and completed during the other flushes and waits during the unmount process....
Quoted 3 lines
Anyway, the patch looks good to me, but I wonder if we should abstract the
li_lsnhandling a bit more to avoid easily running into this kind of problems.
Worth considering, IMO.
Cheers,
Dave.
Signature · 7 lines
Dave Chinner [email protected] _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs