Skip to contentSkip to conversation

xfs: don't eat an EIO/ENOSPC writeback error when scrubbing data fork

PATCH Superseded Darrick J. Wong 2 participants 8 messages Jun 22, 2020 – Jun 23, 2020 XFS filesystem

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: don't eat an EIO/ENOSPC writeback error when scrubbing data fork

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +9 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;
 	}
 
#1 lore raw
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

--D

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 54 lines · Darrick J. Wong
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

Quoted 11 lines

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

Ok. Does scrub deal with left-over delalloc extents correctly in this case?

Cheers,

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw
Quoted 67 lines · Dave Chinner
Quoted 54 lines · Darrick J. Wong
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

Quoted 11 lines

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

Quoted 2 lines

Ok. Does scrub deal with left-over delalloc extents correctly in this case?

It ignores the ones in the incore extent tree and flags them if they show up in the ondisk metadata.

--D

Quoted 4 lines

Cheers,

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw
Quoted 68 lines · Darrick J. Wong
Quoted 67 lines · Dave Chinner
Quoted 65 lines · Darrick J. Wong
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

Quoted 5 lines
Quoted 2 lines

Ok. Does scrub deal with left-over delalloc extents correctly in this case?

It ignores the ones in the incore extent tree and flags them if they show up in the ondisk metadata.

OK. Perhaps add this to the comment?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#6 lore raw
Quoted 69 lines · Dave Chinner
Quoted 68 lines · Darrick J. Wong
Quoted 67 lines · Dave Chinner
Quoted 65 lines · Darrick J. Wong
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Ok. Does scrub deal with left-over delalloc extents correctly in this case?

It ignores the ones in the incore extent tree and flags them if they show up in the ondisk metadata.

OK. Perhaps add this to the comment?

Will do.

--D

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw
In reply to Dave Chinner
Quoted 69 lines · Dave Chinner
Quoted 68 lines · Darrick J. Wong
Quoted 67 lines · Dave Chinner
Quoted 65 lines · Darrick J. Wong
Quoted 48 lines · Dave Chinner
Quoted 32 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

The data fork scrubber calls filemap_write_and_wait to flush dirty pages and delalloc reservations out to disk prior to checking the data fork's extent mappings. Unfortunately, this means that scrub can consume the EIO/ENOSPC errors that would otherwise have stayed around in the address space until (we hope) the writer application calls fsync to persist data and collect errors. The end result is that programs that wrote to a file might never see the error code and proceed as if nothing were wrong.

xfs_scrub is not in a position to notify file writers about the writeback failure, and it's only here to check metadata, not file contents. Therefore, if writeback fails, we should stuff the error code back into the address space so that an fsync by the writer application can pick that up.

Fixes 99d9d8d05da2 ("xfs: scrub inode block mappings")
---
 fs/xfs/scrub/bmap.c |   10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/xfs/scrub/bmap.c +0 -1
index 7badd6dfe544..03be7cf3fe5a 100644
@@ -47,7 +47,15 @@ xchk_setup_inode_bmap(
 	    sc->sm->sm_type == XFS_SCRUB_TYPE_BMBTD) {
 		inode_dio_wait(VFS_I(sc->ip));
 		error = filemap_write_and_wait(VFS_I(sc->ip)->i_mapping);
-		if (error)

fs/xfs/scrub/bmap.c xchk_setup_inode_bmap( · line 59

Quoted 10 lines
fs/xfs/scrub/bmap.c +9 -0
+		if (error == -ENOSPC || error == -EIO) {
+			/*
+			 * If writeback hits EIO or ENOSPC, reflect it back
+			 * into the address space mapping so that a writer
+			 * program calling fsync to look for errors will still
+			 * capture the error.
+			 */
+			mapping_set_error(VFS_I(sc->ip)->i_mapping, error);
+		} else if (error)
 			goto out;

calling mapping_set_error() seems reasonable here and you've explained that well, but shouldn't the error then be processed the same way as all other errors? i.e. by jumping to out?

Quoted 2 lines

If we are now continuing to scrub the bmap after ENOSPC/EIO occur, why?

Heh, ok, more explanation is needed. How about this?

/*
* If writeback hits EIO or ENOSPC, reflect it back into the
* address space mapping so that a writer program calling fsync
* to look for errors will still capture the error. *
* However, we continue into the extent mapping checks because
* write failures do not necessarily imply anything about the
* correctness of the file metadata. The metadata and the file
* data could be on completely separate devices; a media failure
* might only affect a subset of the disk, etc.
*/

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Ok. Does scrub deal with left-over delalloc extents correctly in this case?

It ignores the ones in the incore extent tree and flags them if they show up in the ondisk metadata.

OK. Perhaps add this to the comment?

ok, will do

--D

Quoted 6 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]