commit b4678df184b causing xfstests regressions
Subsystems (3)
Hi Matthew,
Commit b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revert b4678df184b.
Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:
dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null
creating a new loop device
loop_dev=$(_create_loop_device $img_file)
... and then run mke2fs on that loop device.
The instrumentation of mke2fs shows that the first fsync() on /dev/loop0 (in lib/ext2fs/closefs.c) which is failing with EIO.
I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using dm_error, or some such), and somehow the errseq_t for the loop device isn't getting reset, or the inode for the underlying backing file, had an unitialized errseq_t.
Can you take a closer look at this?
Thanks,
- Ted
Quoted 6 lines · Theodore Y. Ts'o
Hi Matthew,
Commit
b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revertb4678df184b.
Thanks; I'll take a look. Monday is a holiday in Canada, so it may be Tuesday before I get to it.
Quoted 29 lines
Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:
dd if=/dev/zero of=$img_file bs=1M count=300 &>
/dev/nullcreating a new loop device
loop_dev=$(_create_loop_device $img_file)
... and then run mke2fs on that loop device.
The instrumentation of mke2fs shows that the first
fsync()on/dev/loop0(inlib/ext2fs/closefs.c) which is failing with EIO.I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using
dm_error, or some such), and somehow theerrseq_tfor the loop device isn't getting reset, or the inode for the underlying backing file, had an unitializederrseq_t.Can you take a closer look at this?
Thanks,
- Ted
Quoted 29 lines · Theodore Y. Ts'o
Hi Matthew,
Commit
b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revertb4678df184b.Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:
dd if=/dev/zero of=$img_file bs=1M count=300 &>
/dev/nullcreating a new loop device
loop_dev=$(_create_loop_device $img_file)
... and then run mke2fs on that loop device.
The instrumentation of mke2fs shows that the first
fsync()on/dev/loop0(inlib/ext2fs/closefs.c) which is failing with EIO.I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using
dm_error, or some such), and somehow theerrseq_tfor the loop device isn't getting reset, or the inode for the underlying backing file, had an unitializederrseq_t.
Quoted 6 lines
Can you take a closer look at this?
Thanks,
- Ted
Thanks Ted. I'm not that familiar with the loopdev code, but after giving it a quick look, I suspect that you're correct. We probably need to do something like reset the loop device's bd_inode->i_mapping->wb_err back to zero when we detach the file that backs it.
I wonder if we could roll a test that would do:
create a scratch fs on a dm-error dev with a file on it set up a loop device on that file
have the backing device of the scratch file throw errors write to the device
detach loop device
clear dm-error condition
delete file and recreate it
attach same loop device to new file
fsync loop device
My suspicion is that that last fsync would throw an error now and it wouldn't have before.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 54 lines · Jeff Layton
Quoted 29 lines · Theodore Y. Ts'o
Hi Matthew,
Commit
b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revertb4678df184b.Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:
dd if=/dev/zero of=$img_file bs=1M count=300 &>
/dev/nullcreating a new loop device
loop_dev=$(_create_loop_device $img_file)
... and then run mke2fs on that loop device.
The instrumentation of mke2fs shows that the first
fsync()on/dev/loop0(inlib/ext2fs/closefs.c) which is failing with EIO.I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using
dm_error, or some such), and somehow theerrseq_tfor the loop device isn't getting reset, or the inode for the underlying backing file, had an unitializederrseq_t.Quoted 6 lines
Can you take a closer look at this?
Thanks,
- Ted
Thanks Ted. I'm not that familiar with the loopdev code, but after giving it a quick look, I suspect that you're correct. We probably need to do something like reset the loop device's
bd_inode->i_mapping->wb_errback to zero when we detach the file that backs it.I wonder if we could roll a test that would do:
create a scratch fs on a dm-error dev with a file on it set up a loop device on that file
have the backing device of the scratch file throw errors write to the device
detach loop device
clear dm-error condition
delete file and recreate it
attach same loop device to new file
fsync loop device
Quoted 2 lines
My suspicion is that that last fsync would throw an error now and it wouldn't have before.
I /think/ it's because inode_init_always doesn't clear mapping->wb_err (even though it clears mapping->flags) when recycling struct inodes. Will send patch shortly.
--D
Quoted 2 lines
Signature · 1 line
Jeff Layton <[email protected]>
PATCH 1/1 fs: clear writeback errors in inode_init_always
In reply to Jeff LaytonFrom: Darrick J. Wong <[email protected]>
In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.
This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.
--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 2 ++ 2 files changed, 3 insertions(+)
fs/inode.c +1 -0
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;
fs/xfs/xfs_icache.c +2 -0
@@ -298,6 +298,7 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; error =inode_init_always (mp->m_super, inode);
@@ -306,6 +307,7 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err;return error; }
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do not
typo of ENOSPC in case you do a new version
Quoted 11 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Quoted 4 lines
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.
Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?
Quoted 17 lines
+++
b/fs/xfs/xfs_icache.cfs/xfs/xfs_icache.c +2
-0@@ -298,6 +298,7 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; error =inode_init_always (mp->m_super, inode);@@ -306,6 +307,7 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err;return error; }
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do nottypo of ENOSPC in case you do a new version
Fixed, thanks.
Quoted 11 lines
Quoted 11 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Quoted 7 lines
Quoted 4 lines
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Don't you also need to preserve
inode->i_mapping->flagsacross the reinitialisation for the users which aren't yet using ->wb_err?
At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.
--D
Quoted 21 lines
Quoted 10 lines
+++
b/fs/xfs/xfs_icache.cfs/xfs/xfs_icache.c +1
-0@@ -298,6 +298,7 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; error =inode_init_always (mp->m_super, inode);fs/xfs/xfs_icache.c xfs_reinit_inode( · line 313
Quoted 7 lines
fs/xfs/xfs_icache.c +1
-0@@ -306,6 +307,7 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err;return error; }--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do nottypo of ENOSPC in case you do a new version
Fixed, thanks.
Quoted 16 lines
Quoted 15 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.
Quoted 9 lines
Quoted 2 lines
Don't you also need to preserve
inode->i_mapping->flagsacross the reinitialisation for the users which aren't yet using ->wb_err?At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old
filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.
Yes, I think the patch you sent makes sense.
Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).
We could (in principle) pass a pointer to file->f_wb_err to most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.
Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
+++
b/fs/xfs/xfs_icache.cfs/xfs/xfs_icache.c +1
-0@@ -298,6 +298,7 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; error =inode_init_always (mp->m_super, inode);fs/xfs/xfs_icache.c xfs_reinit_inode( · line 313
Quoted 7 lines
fs/xfs/xfs_icache.c +1
-0@@ -306,6 +307,7 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err;return error; }--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do nottypo of ENOSPC in case you do a new version
Fixed, thanks.
Quoted 16 lines
Quoted 15 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Quoted 9 lines
Quoted 2 lines
Don't you also need to preserve
inode->i_mapping->flagsacross the reinitialisation for the users which aren't yet using ->wb_err?At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old
filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.Yes, I think the patch you sent makes sense.
Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).
Quoted 5 lines
We could (in principle) pass a pointer to
file->f_wb_errto most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.
Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to fsync() failing and reporting the error to userspace?
This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on fsync(), but can be reported multiple times through different filesystem operations...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 51 lines · Dave Chinner
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do nottypo of ENOSPC in case you do a new version
Fixed, thanks.
Quoted 16 lines
Quoted 15 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Quoted 9 lines
Quoted 2 lines
Don't you also need to preserve
inode->i_mapping->flagsacross the reinitialisation for the users which aren't yet using ->wb_err?At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old
filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.Yes, I think the patch you sent makes sense.
Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).
Quoted 5 lines
We could (in principle) pass a pointer to
file->f_wb_errto most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.
Quoted 9 lines
Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to
fsync()failing and reporting the error to userspace?This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on
fsync(), but can be reported multiple times through different filesystem operations...
If I get back an error on ioctl, why should I think that that has anything to do with writeback? For example, suppose I do:
write = 0
ioctl = -EIO
fsync = 0
The ioctl returns an error but the fsync returns 0. It would be reasonable to assume that my writes had made it to disk.
I think that we really do want to ensure that fsync always returns an error if writeback failed since the last fsync. That was sort of the point of the errseq_t work in the first place.
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 69 lines · Jeff Layton
Quoted 51 lines · Dave Chinner
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do nottypo of ENOSPC in case you do a new version
Fixed, thanks.
Quoted 16 lines
Quoted 15 lines
also clear
wb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Quoted 9 lines
Quoted 2 lines
Don't you also need to preserve
inode->i_mapping->flagsacross the reinitialisation for the users which aren't yet using ->wb_err?At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old
filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.Yes, I think the patch you sent makes sense.
Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).
Quoted 5 lines
We could (in principle) pass a pointer to
file->f_wb_errto most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.Quoted 9 lines
Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to
fsync()failing and reporting the error to userspace?This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on
fsync(), but can be reported multiple times through different filesystem operations...If I get back an error on ioctl, why should I think that that has anything to do with writeback? For example, suppose I do:
write = 0
ioctl =-EIO
fsync = 0
Quoted 6 lines
The ioctl returns an error but the fsync returns 0. It would be reasonable to assume that my writes had made it to disk.
I think that we really do want to ensure that fsync always returns an error if writeback failed since the last fsync. That was sort of the point of the
errseq_twork in the first place.
This needs documentation explaining all these intricacies. Because I'm certain that we'll get it wrong again if the expected behaviour isn't all spelled out very clearly.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH] fs: clear writeback errors in inode_init_always
In reply to Darrick J. WongQuoted 15 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Quoted 6 lines
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>
This may fix the generic/047 failure, but alas, it does not address the shared/298 failure.
Jeff's theory that we may need to clear the errseq_t information after detaching the loop device makes sense to me; and in the case of the loop device, we wouldn't be initializing the inode, so your patch wouldn't do anything about that particular case.
- Ted
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 23 lines · Theodore Y. Ts'o
Quoted 15 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.
Quoted 6 lines
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>
Quoted 9 lines
This may fix the generic/047 failure, but alas, it does not address the shared/298 failure.
Jeff's theory that we may need to clear the
errseq_tinformation after detaching the loop device makes sense to me; and in the case of the loop device, we wouldn't be initializing the inode, so your patch wouldn't do anything about that particular case.
I poked at this a bit this morning and found that if I ran generic/361 twice in a row that I'd see a failure similar to what you were seeing. This patch seems to fix it for me.
Ted, could you test this against your reproducer? If this works for you I'll plan to flesh out the patch description and get this into -next and eventually to Linus before the next release.
Thanks,
Jeff
-------------------------8<--------------------
[PATCH] loop: clear wb_err in bd_inode when detaching backing file
--- drivers/block/loop.c | 1 + 1 file changed, 1 insertion(+)
drivers/block/loop.c +1 -0
@@ -1068,6 +1068,7 @@ static int loop_clr_fd(struct loop_device *lo)if (bdev) {bdput (bdev);invalidate_bdev (bdev); + bdev->bd_inode->i_mapping->wb_err =0 ; }set_capacity (lo->lo_disk,0 );loop_sysfs_exit (lo);
Signature · 1 line
2.17.0
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 1 line · Jeff Layton
[PATCH] loop: clear
wb_errinbd_inodewhen detaching backing file
Is this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.
I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton
[PATCH] loop: clear
wb_errinbd_inodewhen detaching backing fileIs this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.
I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.
This is a bit of a grey area, for sure. FWIW, I looked at doing this in invalidate_bdev, but backed off since I wasn't sure about the other callers.
I still think it probably is the right thing to do. I'd think that detaching the backing store is sort of an indicator that you really don't care about it anymore (including any stored errors concerning it).
OTOH, my track record on predicting what applications care about is pretty abysmal. I'll go with whatever the consensus is here...
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH] fs: clear writeback errors in inode_init_always
In reply to Matthew WilcoxQuoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton
[PATCH] loop: clear
wb_errinbd_inodewhen detaching backing fileIs this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.
I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.
You can't detach the backing store if there are any open file descriptors (or if there is another loop device keeping the loop device open, or if there is a file system mounted on it, etc.).
If you can detach the backing store, once you detach the backing store, the loop device is *gone*. The user of /dev/loop0 will very likely be a completely different backing store, so returning an error simply doesn't make any sense. There is a very good chance it will be a completely different backing store, with potentially a different user, so returning a carried over error is just going confuse, annoy, and anger the user.....
- Ted
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 17 lines · Theodore Y. Ts'o
Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton
[PATCH] loop: clear
wb_errinbd_inodewhen detaching backing fileIs this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.
I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.
You can't detach the backing store if there are any open file descriptors (or if there is another loop device keeping the loop device open, or if there is a file system mounted on it, etc.).
Quoted 7 lines
If you can detach the backing store, once you detach the backing store, the loop device is *gone*. The user of
/dev/loop0will very likely be a completely different backing store, so returning an error simply doesn't make any sense. There is a very good chance it will be a completely different backing store, with potentially a different user, so returning a carried over error is just going confuse, annoy, and anger the user.....
Oh! I misunderstood. I thought that bd_inode->i_mapping pointed to lo_backing_file->f_mapping.
So how about this to preserve the error _in the file with the error_?
if (bdev) {
bdput(bdev);
invalidate_bdev(bdev);
+ filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err;
+ bdev->bd_inode->i_mapping->wb_err = 0;
}
set_capacity(lo->lo_disk, 0);
loop_sysfs_exit(lo);Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 10 lines · Matthew Wilcox
Oh! I misunderstood. I thought that
bd_inode->i_mappingpointed tolo_backing_file->f_mapping.So how about this to preserve the error
_inthe file with theerror_?if (bdev) { bdput(bdev); invalidate_bdev(bdev);(quoted diff) +2
-0+ filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err; + bdev->bd_inode->i_mapping->wb_err = 0;
Quoted 3 lines
} set_capacity(lo->lo_disk, 0); loop_sysfs_exit(lo);
I don't think it's necessary. wb_err on the underlying file would have already been set when the error was set. So if someone tried calling fsync(2) on the underlying file, it would have collected the error already. Re-setting the error when we detach the loop device doesn't seem to make any sense.
- Ted
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 6 lines · Theodore Y. Ts'o
Quoted 5 lines · Matthew Wilcox
Oh! I misunderstood. I thought that
bd_inode->i_mappingpointed tolo_backing_file->f_mapping.So how about this to preserve the error
_inthe file with theerror_?
Quoted 15 lines
Quoted 8 lines
if (bdev) { bdput(bdev); invalidate_bdev(bdev);(quoted diff) +2
-0+ filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err; + bdev->bd_inode->i_mapping->wb_err = 0;} set_capacity(lo->lo_disk, 0); loop_sysfs_exit(lo);I don't think it's necessary.
wb_erron the underlying file would have already been set when the error was set. So if someone tried calling fsync(2) on the underlying file, it would have collected the error already. Re-setting the error when we detach the loop device doesn't seem to make any sense.
(cc'ing linux-block)
Yes, the error would have been set on the original file already. As long as lo_req_flush or __loop_update_dio weren't called before we detach the device, it should still be possible to call fsync on the original fd and get back the error.
As a side note, it looks like __loop_update_dio will discard an error from vfs_fsync, so certain ioctls against a loop device can cause errors to be lost. It seems like those ought to get propagated to userland or to the blockdev's mapping somehow.
Signature · 1 line
Jeff Layton <[email protected]>
Re: [PATCH] fs: clear writeback errors in inode_init_always
Quoted 5 lines · Jeff Layton
As a side note, it looks like
__loop_update_diowill discard an error fromvfs_fsync, so certain ioctls against a loop device can cause errors to be lost. It seems like those ought to get propagated to userland or to the blockdev's mapping somehow.
I'm not sure it's worth it. All of the ioctls that call loop_update_dio are used just to configure the loop device. In practice they are never called once the loop device is actually running. It might be a better choice is to just define that errors get reset if there is any attempt to reconfigure the loop device.
One of these ioctls change the offset that a loop device maps onto the backing file. So for example, while offset 0 of the loop device might have corresponds beginning of the backing file, after executing the ioctl, offset 0 of the loop device no corresponds to offset 2MB of the backing store. This might happen if we are resetting the loop device to point at the first partition of a disk image, for example. I really don't think that preserving the error status when we are doing that kind of radical configuration makes sense.
Or for example, when we change the block size of the loop device; if the underlying backing store is an IBM Mainframe DASD with a 2k block size, the reason why the error was signalled was because there was an attempt to write a 1k block onto a 2k block device. So again, resetting the error status of the loop device is the right thing to do.
The final thing that's worth perhaps exploring is whether or not we need all of these knobs and ioctls. In particular, LOOP_CHANGE_FD is used by losetup. It exists because back in prehistory, some distribution installer needed it for some obscure corner case. If I remember correctly, it had to do with switching out the initramfs floppy disk with the root file system floppy disk. We've wasted developer bandwidth trying to deal with syzbot complaints about that ioctl, and it's not clear it was worth the effort, because it's not clear any real code actually _needs_ that ioctl. It might have been easier to comment out the ioctl, and see if anyone screamed.
Given that loop device is maintianed mostly on the margins, and it's not clear that lots of complicated error handling during setup is really necessary, this is basically a please to keep things simple, or at least no more complicated than it has to be.
Cheers,
- Ted
Re: [PATCH] fs: clear writeback errors in inode_init_always
In reply to Jeff LaytonQuoted 8 lines · Jeff Layton
I poked at this a bit this morning and found that if I ran generic/361 twice in a row that I'd see a failure similar to what you were seeing. This patch seems to fix it for me.
Ted, could you test this against your reproducer? If this works for you I'll plan to flesh out the patch description and get this into -next and eventually to Linus before the next release.
Yup, this fixes shared/298 for me. Many thanks!!
- Ted
PATCH v2 1/1 fs: clear writeback errors in inode_init_always
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.
This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.
---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit
--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ 2 files changed, 10 insertions(+)
fs/inode.c +1 -0
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;
fs/xfs/xfs_icache.c +9 -0
@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);
@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);return error; }
Re: [PATCH v2] fs: clear writeback errors in inode_init_always
Quoted 57 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ 2 files changed, 10 insertions(+)fs/inode.c +1
-0@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;fs/xfs/xfs_icache.c +4
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version);
fs/xfs/xfs_icache.c xfs_reinit_inode( · line 320
Quoted 10 lines
fs/xfs/xfs_icache.c +5
-0inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);return error; }
Re: [PATCH v2] fs: clear writeback errors in inode_init_always
In reply to Darrick J. WongQuoted 28 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ 2 files changed, 10 insertions(+)
...
Quoted 16 lines
fs/xfs/xfs_icache.c +4
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);
fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317
Quoted 8 lines
fs/xfs/xfs_icache.c +5
-0@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);
I'm wondering how safe this is. Can't the associated on-disk inode have been unlinked and reallocated anew across this kind of reinit of the in-core ip?
Brian
Quoted 7 lines
fs/xfs/xfs_icache.c +0
-1return error; } --To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] fs: clear writeback errors in inode_init_always
Quoted 45 lines · Brian Foster
Quoted 28 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ 2 files changed, 10 insertions(+)...
Quoted 15 lines
fs/xfs/xfs_icache.c +4
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);
Quoted 13 lines
fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317
Quoted 9 lines
fs/xfs/xfs_icache.c +5
-0@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);I'm wondering how safe this is. Can't the associated on-disk inode have been unlinked and reallocated anew across this kind of reinit of the in-core ip?
Oops, yeah, xfs_ialloc ought to clear those error states unconditionally when allocating a new on-disk inode.
--D
Quoted 13 lines
Brian
Quoted 7 lines
fs/xfs/xfs_icache.c +0
-1return error; } --To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v3 1/1 fs: clear writeback errors in inode_init_always
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.
This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.
---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit
--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ fs/xfs/xfs_inode.c | 5 +++++ 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;
fs/xfs/xfs_icache.c +9 -0
@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);
@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);return error; }
fs/xfs/xfs_inode.c +5 -0
@@ -835,6 +835,11 @@ xfs_ialloc( inode->i_mode |=S_ISGID ; } +/* Reset all vfs error state. */ + inode->i_mapping->wb_err =0 ; +clear_bit (AS_EIO , &inode->i_mapping->flags); +clear_bit (AS_ENOSPC , &inode->i_mapping->flags); +/* * If the group ID of the new file does not match the effective group * ID or one of the supplementary group IDs, the S_ISGID bit is cleared
Re: [PATCH v3] fs: clear writeback errors in inode_init_always
Quoted 75 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Jeff Layton <[email protected]>---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ fs/xfs/xfs_inode.c | 5 +++++ 3 files changed, 15 insertions(+)fs/inode.c +1
-0@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;fs/xfs/xfs_icache.c +9
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);return error; }fs/xfs/xfs_inode.c +0
-0
fs/xfs/xfs_inode.c xfs_ialloc( · line 838
Quoted 4 lines
fs/xfs/xfs_inode.c +1
-0@@ -835,6 +835,11 @@ xfs_ialloc( inode->i_mode |=S_ISGID ; } +/* Reset all vfs error state. */
I'd prefer a comment that reminds why we have to do this. E.g., something like:
/*
* In-core inode reinit carries over vfs error state. Reset it
* so we don't leak error state from a previous generation of
* the inode.
*/
... but feel free to reword that.
fs/xfs/xfs_inode.c xfs_ialloc( · line 842
Quoted 4 lines
fs/xfs/xfs_inode.c +4
-0+ inode->i_mapping->wb_err =0 ; +clear_bit (AS_EIO , &inode->i_mapping->flags); +clear_bit (AS_ENOSPC , &inode->i_mapping->flags); +
I also wonder whether it might be cleaner to just reset this stuff in xfs_ifree() where we kill the i_mode and bump the gen and whatnot, but afaict this should work too. With the comment fix, at least:
Quoted 7 lines
fs/xfs/xfs_inode.c +0
-1/* * If the group ID of the new file does not match the effective group * ID or one of the supplementary group IDs, the S_ISGID bit is cleared -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] fs: clear writeback errors in inode_init_always
Quoted 84 lines · Brian Foster
Quoted 75 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Jeff Layton <[email protected]>---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ fs/xfs/xfs_inode.c | 5 +++++ 3 files changed, 15 insertions(+)fs/inode.c +1
-0@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;fs/xfs/xfs_icache.c +9
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);return error; }fs/xfs/xfs_inode.c +0
-0fs/xfs/xfs_inode.c xfs_ialloc( · line 838
Quoted 4 lines
fs/xfs/xfs_inode.c +1
-0@@ -835,6 +835,11 @@ xfs_ialloc( inode->i_mode |=S_ISGID ; } +/* Reset all vfs error state. */ I'd prefer a comment that reminds why we have to do this. E.g., something like:
Quoted 5 lines
/*
* In-core inode reinit carries over vfs error state. Reset it
* so we don't leak error state from a previous generation of
* the inode.
*/
I settled on:
/*
* In-core inode reinit does not clear vfs error state. Reset
* it so we don't leak error state from a previous generation of
* the inode.
*/
Quoted 12 lines
... but feel free to reword that.
fs/xfs/xfs_inode.c xfs_ialloc( · line 842
Quoted 4 lines
fs/xfs/xfs_inode.c +4
-0+ inode->i_mapping->wb_err =0 ; +clear_bit (AS_EIO , &inode->i_mapping->flags); +clear_bit (AS_ENOSPC , &inode->i_mapping->flags); +I also wonder whether it might be cleaner to just reset this stuff in
xfs_ifree()where we kill thei_modeand bump the gen and whatnot, but afaict this should work too. With the comment fix, at least:
Quoted 1 line
Reviewed-by Brian Foster <[email protected]>
Thanks for the review!
--D
Quoted 12 lines
Quoted 8 lines
fs/xfs/xfs_inode.c +0
-1/* * If the group ID of the new file does not match the effective group * ID or one of the supplementary group IDs, the S_ISGID bit is cleared -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] fs: clear writeback errors in inode_init_always
In reply to Darrick J. WongQuoted 59 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Jeff Layton <[email protected]>---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ fs/xfs/xfs_inode.c | 5 +++++ 3 files changed, 15 insertions(+)fs/inode.c +1
-0@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;fs/xfs/xfs_icache.c +4
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);
fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317
Quoted 8 lines
fs/xfs/xfs_icache.c +5
-0@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);
I just don't see this as valid.
The inode has already been removed from the VFS cache when this happens (i.e. it's gone through iput_final()->evict->destroy_inode) and so any VFS-specific state is already invalid.
We're retaining XFS-specific inode information across the recycling of the xfs_inode (i.e. stuff that doesn't change *on-disk* across recycling an inode, but is reset by inode_init_always()). This is a creating a new life cycle for the VFS inode (i.e. I_NEW is set), so *there is no VFS state retained* across this operation from it's previous life.
Trying to do this is just going to lead us into nasty layering violations where the VFS state has to be hacked around just in case this recycling of the previous life cycle's state was the wrong thing to do.
fs/xfs/xfs_inode.c xfs_ialloc( · line 841
Quoted 12 lines
fs/xfs/xfs_inode.c +4
-0@@ -835,6 +835,11 @@ xfs_ialloc( inode->i_mode |=S_ISGID ; } +/* Reset all vfs error state. */ + inode->i_mapping->wb_err =0 ; +clear_bit (AS_EIO , &inode->i_mapping->flags); +clear_bit (AS_ENOSPC , &inode->i_mapping->flags);
Just like this. This is a sure sign we're doing the wrong thing in xfs_reinit_inode(). VFS inode lifecycle state is trashed at ->destroy_inode. It does not persist into new instantiations of VFS inode state, regardless of that new inode lifecycle gets to the I_NEW state...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v3] fs: clear writeback errors in inode_init_always
Quoted 95 lines · Dave Chinner
Quoted 59 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
In
inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO,AS_ENOSPC) bits. Unfortunately, we do not also clearwb_err, which means that old mapping errors can leak through to new inodes.This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.
Since we're changing the semantics of
inode_init_always, we must also changexfs_reinit_inodeto retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.Signed-off-by Darrick J. Wong <[email protected]>Reviewed-by Jeff Layton <[email protected]>---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit--- fs/inode.c | 1 + fs/xfs/xfs_icache.c | 9 +++++++++ fs/xfs/xfs_inode.c | 5 +++++ 3 files changed, 15 insertions(+)fs/inode.c +1
-0@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags =0 ; + mapping->wb_err =0 ;atomic_set (&mapping->i_mmap_writable,0 );mapping_set_gfp_mask (mapping,GFP_HIGHUSER_MOVABLE ); mapping->private_data =NULL ;fs/xfs/xfs_icache.c +4
-0@@ -298,6 +298,10 @@ xfs_reinit_inode(uint64_t version =inode_peek_iversion (inode);umode_t mode = inode->i_mode;dev_t dev = inode->i_rdev; +errseq_t old_err = inode->i_mapping->wb_err; +bool as_eio =test_bit (AS_EIO , &inode->i_mapping->flags); +bool as_enospc =test_bit (AS_ENOSPC , + &inode->i_mapping->flags); error =inode_init_always (mp->m_super, inode);fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317
Quoted 8 lines
fs/xfs/xfs_icache.c +5
-0@@ -306,6 +310,11 @@ xfs_reinit_inode(inode_set_iversion_queried (inode, version); inode->i_mode = mode; inode->i_rdev = dev; + inode->i_mapping->wb_err = old_err; +if (as_eio) +set_bit (AS_EIO , &inode->i_mapping->flags); +if (as_enospc) +set_bit (AS_ENOSPC , &inode->i_mapping->flags);I just don't see this as valid.
The inode has already been removed from the VFS cache when this happens (i.e. it's gone through iput_final()->evict->destroy_inode) and so any VFS-specific state is already invalid.
We're retaining XFS-specific inode information across the recycling of the
xfs_inode(i.e. stuff that doesn't change *on-disk* across recycling an inode, but is reset byinode_init_always()). This is a creating a new life cycle for the VFS inode (i.e.I_NEWis set), so *there is no VFS state retained* across this operation from it's previous life.Trying to do this is just going to lead us into nasty layering violations where the VFS state has to be hacked around just in case this recycling of the previous life cycle's state was the wrong thing to do.
Quoted 8 lines
fs/xfs/xfs_inode.c +0
-0@@ -835,6 +835,11 @@ xfs_ialloc( inode->i_mode |=S_ISGID ; }
Quoted 10 lines
fs/xfs/xfs_inode.c xfs_ialloc( · line 841
Quoted 4 lines
fs/xfs/xfs_inode.c +4
-0+/* Reset all vfs error state. */ + inode->i_mapping->wb_err =0 ; +clear_bit (AS_EIO , &inode->i_mapping->flags); +clear_bit (AS_ENOSPC , &inode->i_mapping->flags);Just like this. This is a sure sign we're doing the wrong thing in
xfs_reinit_inode(). VFS inode lifecycle state is trashed at ->destroy_inode. It does not persist into new instantiations of VFS inode state, regardless of that new inode lifecycle gets to theI_NEWstate...
(Just reiterating a conversation we had on irc...)
So basically as far as the vfs is concerned the inode is gone and done and destroyed, and it has no expectation of state being preserved. Therefore, this patch can be trimmed to the inode_init_always part.
--D
Quoted 10 lines
Cheers,
Dave.
Signature · 6 lines
Dave Chinner [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html