Skip to contentSkip to conversation

xfs: copy li_lsn before dropping AIL lock

PATCH In review Nathaniel W. Turner 3 participants 3 messages Nov 16, 2009 – Nov 17, 2009 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: copy li_lsn before dropping AIL lock

Access to log items on the AIL is generally protected by m_ail_lock; this is particularly needed when we're getting or setting the 64-bit li_lsn on a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.

This can result in a partially-zeroed log->l_tail_lsn if xfs_trans_ail_delete is racing with xfs_trans_ail_update, and in at least some cases, this can leave the l_tail_lsn with a zero cycle number, which means xlog_space_left will think the log is full (unless CONFIG_XFS_DEBUG is set, in which case we'll trip an ASSERT), leading to processes stuck forever in xlog_grant_log_space.

Thanks to Adrian VanderSpek for first spotting the race potential and to Dave Chinner for debug assistance.

---
 fs/xfs/xfs_trans_ail.c |   23 ++++++++++++++++++++---
 1 files changed, 20 insertions(+), 3 deletions(-)
fs/xfs/xfs_trans_ail.c +20 -3
index f31271c..2ffc570 100644
@@ -467,6 +467,7 @@ xfs_trans_ail_update(
 {
 	xfs_log_item_t		*dlip = NULL;
 	xfs_log_item_t		*mlip;	/* ptr to minimum lip */
+	xfs_lsn_t		tail_lsn;
 
 	mlip = xfs_ail_min(ailp);
 
@@ -483,8 +484,16 @@ xfs_trans_ail_update(
 
 	if (mlip == dlip) {
 		mlip = xfs_ail_min(ailp);
+		/*
+		 * It is not safe to access mlip after the AIL lock is
+		 * dropped, so we must get a copy of li_lsn before we do
+		 * so.  This is especially important on 32-bit platforms
+		 * where accessing and updating 64-bit values like li_lsn
+		 * is not atomic.
+		 */
+		tail_lsn = mlip->li_lsn;
 		spin_unlock(&ailp->xa_lock);
-		xfs_log_move_tail(ailp->xa_mount, mlip->li_lsn);
+		xfs_log_move_tail(ailp->xa_mount, tail_lsn);
 	} else {
 		spin_unlock(&ailp->xa_lock);
 	}
@@ -514,6 +523,7 @@ xfs_trans_ail_delete(
 {
 	xfs_log_item_t		*dlip;
 	xfs_log_item_t		*mlip;
+	xfs_lsn_t		tail_lsn;
 
 	if (lip->li_flags & XFS_LI_IN_AIL) {
 		mlip = xfs_ail_min(ailp);
@@ -527,9 +537,16 @@ xfs_trans_ail_delete(
 
 		if (mlip == dlip) {
 			mlip = xfs_ail_min(ailp);
+			/*
+			 * It is not safe to access mlip after the AIL lock
+			 * is dropped, so we must get a copy of li_lsn
+			 * before we do so.  This is especially important
+			 * on 32-bit platforms where accessing and updating
+			 * 64-bit values like li_lsn is not atomic.
+			 */
+			tail_lsn = mlip ? mlip->li_lsn : 0;
 			spin_unlock(&ailp->xa_lock);
-			xfs_log_move_tail(ailp->xa_mount,
-						(mlip ? mlip->li_lsn : 0));
+			xfs_log_move_tail(ailp->xa_mount, tail_lsn);
 		} else {
 			spin_unlock(&ailp->xa_lock);
 		}
Signature · 7 lines
Nathaniel W. Turner
http://houseofnate.net/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v1 1/1
#1 lore raw
Quoted 6 lines · Nathaniel W. Turner

Access to log items on the AIL is generally protected by m_ail_lock; this is particularly needed when we're getting or setting the 64-bit li_lsn on a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.

Quoted 6 lines

This can result in a partially-zeroed log->l_tail_lsn if xfs_trans_ail_delete is racing with xfs_trans_ail_update, and in at least some cases, this can leave the l_tail_lsn with a zero cycle number, which means xlog_space_left will think the log is full (unless CONFIG_XFS_DEBUG is set, in which case we'll trip an ASSERT), leading to processes stuck forever in xlog_grant_log_space.

Might this also cause this oops?

http://www.kerneloops.org/raw.php?rawid=944396&msgid=

It's been shoving up a few times recently.

Anyway, the patch looks good to me, but I wonder if we should abstract the li_lsn handling a bit more to avoid easily running into this kind of problems.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw
Quoted 14 lines · Christoph Hellwig
Quoted 6 lines · Nathaniel W. Turner

Access to log items on the AIL is generally protected by m_ail_lock; this is particularly needed when we're getting or setting the 64-bit li_lsn on a 32-bit platform. This patch fixes a couple places where we were accessing the log item after dropping the AIL lock on 32-bit machines.

Quoted 6 lines

This can result in a partially-zeroed log->l_tail_lsn if xfs_trans_ail_delete is racing with xfs_trans_ail_update, and in at least some cases, this can leave the l_tail_lsn with a zero cycle number, which means xlog_space_left will think the log is full (unless CONFIG_XFS_DEBUG is set, in which case we'll trip an ASSERT), leading to processes stuck forever in xlog_grant_log_space.

Quoted 5 lines

Might this also cause this oops?

http://www.kerneloops.org/raw.php?rawid=944396&msgid=

It's been shoving up a few times recently.

I don't think so. That trace is in xfs_close_devices(), which is called after xfs_unmountfs->xfs_log_unmount->xlog_dealloc, which tends to imply that mp->m_log is NULL.

i.e. we have IO being flushed when no IO should be pending. Given that there is a xfs->bdstrat_cb->xfs_bioerror trace in there, it looks like the buffer could not be written because the system is in a forced shutdown situation. I have no idea what that buffer might be or why it wasn't issued and completed during the other flushes and waits during the unmount process....

Quoted 3 lines

Anyway, the patch looks good to me, but I wonder if we should abstract the li_lsn handling a bit more to avoid easily running into this kind of problems.

Worth considering, IMO.

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs