Skip to contentSkip to conversation

xfs: fix iclog release error check race with shutdown

PATCH Superseded Brian Foster 3 participants 6 messages Feb 14, 2020 – Feb 18, 2020 XFS filesystem

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: fix iclog release error check race with shutdown

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

---
 fs/xfs/xfs_log.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +4 -0
index f6006d94a581..f38fc492a14d 100644
@@ -611,6 +611,10 @@ xfs_log_release_iclog(
 	}
 
 	if (atomic_dec_and_lock(&iclog->ic_refcnt, &log->l_icloglock)) {
+		if (iclog->ic_state == XLOG_STATE_IOERROR) {
+			spin_unlock(&log->l_icloglock);
+			return -EIO;
+		}
 		sync = __xlog_state_release_iclog(log, iclog);
 		spin_unlock(&log->l_icloglock);
 		if (sync)
Signature · 1 line
2.21.1
#1 lore raw
Quoted 15 lines · Brian Foster

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Quoted 6 lines

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

On vacation* today so not thinking hard about reviews but if this goes in, Darrick can you please add a:

Fixes df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held")

Thanks,
-Eric

#2 lore raw
In reply to Brian Foster
Quoted 30 lines · Brian Foster

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

---
 fs/xfs/xfs_log.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +0 -0
index f6006d94a581..f38fc492a14d 100644

fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617

Quoted 7 lines
fs/xfs/xfs_log.c +4 -0
@@ -611,6 +611,10 @@ xfs_log_release_iclog(
 	}
 
 	if (atomic_dec_and_lock(&iclog->ic_refcnt, &log->l_icloglock)) {
+		if (iclog->ic_state == XLOG_STATE_IOERROR) {
+			spin_unlock(&log->l_icloglock);
+			return -EIO;
+		}

So the check just above also shuts the file system down. Any reason to do that in one case and not the other?

#3 lore raw
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

---
 fs/xfs/xfs_log.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +0 -0
index f6006d94a581..f38fc492a14d 100644
@@ -611,6 +611,10 @@ xfs_log_release_iclog(
 	}
 
Quoted 9 lines

fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617

Quoted 5 lines
fs/xfs/xfs_log.c +4 -0
 	if (atomic_dec_and_lock(&iclog->ic_refcnt, &log->l_icloglock)) {
+		if (iclog->ic_state == XLOG_STATE_IOERROR) {
+			spin_unlock(&log->l_icloglock);
+			return -EIO;
+		}

So the check just above also shuts the file system down. Any reason to do that in one case and not the other?

The initial check (with the shutdown) was originally associated with the return from xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.

Brian

#4 lore raw
Quoted 44 lines · Brian Foster
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

---
 fs/xfs/xfs_log.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +0 -0
index f6006d94a581..f38fc492a14d 100644
@@ -611,6 +611,10 @@ xfs_log_release_iclog(
 	}
 
Quoted 9 lines

fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617

Quoted 5 lines
fs/xfs/xfs_log.c +4 -0
 	if (atomic_dec_and_lock(&iclog->ic_refcnt, &log->l_icloglock)) {
+		if (iclog->ic_state == XLOG_STATE_IOERROR) {
+			spin_unlock(&log->l_icloglock);
+			return -EIO;
+		}

So the check just above also shuts the file system down. Any reason to do that in one case and not the other?

Quoted 6 lines

The initial check (with the shutdown) was originally associated with the return from xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.

The original code did the force shutdown for both cases. So unless we have a good reason to do it differently I'd just add a goto label and merge the two cases to restore the old behavior.

#5 lore raw
Quoted 52 lines · Christoph Hellwig
Quoted 44 lines · Brian Foster
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster

Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.

The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.

Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.

---
 fs/xfs/xfs_log.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +0 -0
index f6006d94a581..f38fc492a14d 100644
@@ -611,6 +611,10 @@ xfs_log_release_iclog(
 	}
 
Quoted 9 lines

fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617

Quoted 5 lines
fs/xfs/xfs_log.c +4 -0
 	if (atomic_dec_and_lock(&iclog->ic_refcnt, &log->l_icloglock)) {
+		if (iclog->ic_state == XLOG_STATE_IOERROR) {
+			spin_unlock(&log->l_icloglock);
+			return -EIO;
+		}

So the check just above also shuts the file system down. Any reason to do that in one case and not the other?

Quoted 6 lines

The initial check (with the shutdown) was originally associated with the return from xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.

Quoted 4 lines

The original code did the force shutdown for both cases. So unless we have a good reason to do it differently I'd just add a goto label and merge the two cases to restore the old behavior.

Ok. I'm not sure I see the point, but it's harmless and I can make Eric's fix as well so I'll post a v2..

Brian