xfs: fix iclog release error check race with shutdown
PATCH 1/1 xfs: fix iclog release error check race with shutdown
Prior to commit df732b29c8 ("xfs: call xlog_state_release_iclog with l_icloglock held"), xlog_state_release_iclog() always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part of xlog_state_release_iclog() was open-coded into xfs_log_release_iclog() and as a result the locked error state check was lost.
The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in __xlog_state_release_iclog() due to an unexpected iclog state.
Restore the locked error state check in xlog_state_release_iclog() to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.
--- fs/xfs/xfs_log.c | 4 ++++ 1 file changed, 4 insertions(+)
fs/xfs/xfs_log.c +4 -0
@@ -611,6 +611,10 @@ xfs_log_release_iclog( }if (atomic_dec_and_lock (&iclog->ic_refcnt, &log->l_icloglock)) { +if (iclog->ic_state ==XLOG_STATE_IOERROR ) { +spin_unlock (&log->l_icloglock); +return -EIO ; + } sync =__xlog_state_release_iclog (log, iclog);spin_unlock (&log->l_icloglock);if (sync)
Signature · 1 line
2.21.1
Quoted 15 lines · Brian Foster
Prior to commit
df732b29c8("xfs: callxlog_state_release_iclogwithl_icloglockheld"),xlog_state_release_iclog()always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part ofxlog_state_release_iclog()was open-coded intoxfs_log_release_iclog()and as a result the locked error state check was lost.The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in
__xlog_state_release_iclog()due to an unexpected iclog state.
Quoted 6 lines
Restore the locked error state check in
xlog_state_release_iclog()to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.Reported-by Zorro Lang <[email protected]>Signed-off-by Brian Foster <[email protected]>
On vacation* today so not thinking hard about reviews but if this goes in, Darrick can you please add a:
Thanks,
-Eric
Quoted 30 lines · Brian Foster
Prior to commit
df732b29c8("xfs: callxlog_state_release_iclogwithl_icloglockheld"),xlog_state_release_iclog()always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part ofxlog_state_release_iclog()was open-coded intoxfs_log_release_iclog()and as a result the locked error state check was lost.The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in
__xlog_state_release_iclog()due to an unexpected iclog state.Restore the locked error state check in
xlog_state_release_iclog()to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.Reported-by Zorro Lang <[email protected]>Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_log.c | 4 ++++ 1 file changed, 4 insertions(+)fs/xfs/xfs_log.c +0
-0
fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617
Quoted 7 lines
fs/xfs/xfs_log.c +4
-0@@ -611,6 +611,10 @@ xfs_log_release_iclog( }if (atomic_dec_and_lock (&iclog->ic_refcnt, &log->l_icloglock)) { +if (iclog->ic_state ==XLOG_STATE_IOERROR ) { +spin_unlock (&log->l_icloglock); +return -EIO ; + }
So the check just above also shuts the file system down. Any reason to do that in one case and not the other?
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster
Prior to commit
df732b29c8("xfs: callxlog_state_release_iclogwithl_icloglockheld"),xlog_state_release_iclog()always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part ofxlog_state_release_iclog()was open-coded intoxfs_log_release_iclog()and as a result the locked error state check was lost.The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in
__xlog_state_release_iclog()due to an unexpected iclog state.Restore the locked error state check in
xlog_state_release_iclog()to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.Reported-by Zorro Lang <[email protected]>Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_log.c | 4 ++++ 1 file changed, 4 insertions(+)fs/xfs/xfs_log.c +0
-0@@ -611,6 +611,10 @@ xfs_log_release_iclog( }
Quoted 9 lines
fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617
Quoted 5 lines
fs/xfs/xfs_log.c +4
-0if (atomic_dec_and_lock (&iclog->ic_refcnt, &log->l_icloglock)) { +if (iclog->ic_state ==XLOG_STATE_IOERROR ) { +spin_unlock (&log->l_icloglock); +return -EIO ; + }So the check just above also shuts the file system down. Any reason to do that in one case and not the other?
The initial check (with the shutdown) was originally associated with the return from xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.
Brian
Quoted 44 lines · Brian Foster
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster
Prior to commit
df732b29c8("xfs: callxlog_state_release_iclogwithl_icloglockheld"),xlog_state_release_iclog()always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part ofxlog_state_release_iclog()was open-coded intoxfs_log_release_iclog()and as a result the locked error state check was lost.The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in
__xlog_state_release_iclog()due to an unexpected iclog state.Restore the locked error state check in
xlog_state_release_iclog()to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.Reported-by Zorro Lang <[email protected]>Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_log.c | 4 ++++ 1 file changed, 4 insertions(+)fs/xfs/xfs_log.c +0
-0@@ -611,6 +611,10 @@ xfs_log_release_iclog( }Quoted 9 lines
fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617
Quoted 5 lines
fs/xfs/xfs_log.c +4
-0if (atomic_dec_and_lock (&iclog->ic_refcnt, &log->l_icloglock)) { +if (iclog->ic_state ==XLOG_STATE_IOERROR ) { +spin_unlock (&log->l_icloglock); +return -EIO ; + }So the check just above also shuts the file system down. Any reason to do that in one case and not the other?
Quoted 6 lines
The initial check (with the shutdown) was originally associated with the return from
xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.
The original code did the force shutdown for both cases. So unless we have a good reason to do it differently I'd just add a goto label and merge the two cases to restore the old behavior.
Quoted 52 lines · Christoph Hellwig
Quoted 44 lines · Brian Foster
Quoted 33 lines · Christoph Hellwig
Quoted 32 lines · Brian Foster
Prior to commit
df732b29c8("xfs: callxlog_state_release_iclogwithl_icloglockheld"),xlog_state_release_iclog()always performed a locked check of the iclog error state before proceeding into the sync state processing code. As of this commit, part ofxlog_state_release_iclog()was open-coded intoxfs_log_release_iclog()and as a result the locked error state check was lost.The lockless check still exists, but this doesn't account for the possibility of a race with a shutdown being performed by another task causing the iclog state to change while the original task waits on ->l_icloglock. This has reproduced very rarely via generic/475 and manifests as an assert failure in
__xlog_state_release_iclog()due to an unexpected iclog state.Restore the locked error state check in
xlog_state_release_iclog()to ensure that an iclog state update via shutdown doesn't race with the iclog release state processing code.Reported-by Zorro Lang <[email protected]>Signed-off-by Brian Foster <[email protected]>--- fs/xfs/xfs_log.c | 4 ++++ 1 file changed, 4 insertions(+)fs/xfs/xfs_log.c +0
-0@@ -611,6 +611,10 @@ xfs_log_release_iclog( }Quoted 9 lines
fs/xfs/xfs_log.c xfs_log_release_iclog( · line 617
Quoted 5 lines
fs/xfs/xfs_log.c +4
-0if (atomic_dec_and_lock (&iclog->ic_refcnt, &log->l_icloglock)) { +if (iclog->ic_state ==XLOG_STATE_IOERROR ) { +spin_unlock (&log->l_icloglock); +return -EIO ; + }So the check just above also shuts the file system down. Any reason to do that in one case and not the other?
Quoted 6 lines
The initial check (with the shutdown) was originally associated with the return from
xlog_state_release_iclog(). That covers both state checks, as they were both originally within that function. My impression was there isn't a need to shutdown in the second check because the only way the iclog state changes to IOERROR across that lock cycle is due to a shutdown already in progress.
Quoted 4 lines
The original code did the force shutdown for both cases. So unless we have a good reason to do it differently I'd just add a goto label and merge the two cases to restore the old behavior.
Ok. I'm not sure I see the point, but it's harmless and I can make Eric's fix as well so I'll post a v2..
Brian