Skip to contentSkip to conversation

commit b4678df184b causing xfstests regressions

PATCH Superseded Theodore Y. Ts'o 6 participants 30 messages May 18, 2018 – May 24, 2018

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi Matthew,

Commit b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revert b4678df184b.

Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:

dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null

creating a new loop device

loop_dev=$(_create_loop_device $img_file)

... and then run mke2fs on that loop device.

The instrumentation of mke2fs shows that the first fsync() on /dev/loop0 (in lib/ext2fs/closefs.c) which is failing with EIO.

I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using dm_error, or some such), and somehow the errseq_t for the loop device isn't getting reset, or the inode for the underlying backing file, had an unitialized errseq_t.

Can you take a closer look at this?

Thanks,

- Ted

#1 lore raw
Quoted 6 lines · Theodore Y. Ts'o

Hi Matthew,

Commit b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revert b4678df184b.

Thanks; I'll take a look. Monday is a holiday in Canada, so it may be Tuesday before I get to it.

Quoted 29 lines

Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:

dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null

creating a new loop device

loop_dev=$(_create_loop_device $img_file)

... and then run mke2fs on that loop device.

The instrumentation of mke2fs shows that the first fsync() on /dev/loop0 (in lib/ext2fs/closefs.c) which is failing with EIO.

I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using dm_error, or some such), and somehow the errseq_t for the loop device isn't getting reset, or the inode for the underlying backing file, had an unitialized errseq_t.

Can you take a closer look at this?

Thanks,

- Ted

#2 lore raw
In reply to Theodore Y. Ts'o
Quoted 29 lines · Theodore Y. Ts'o

Hi Matthew,

Commit b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revert b4678df184b.

Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:

dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null

creating a new loop device

loop_dev=$(_create_loop_device $img_file)

... and then run mke2fs on that loop device.

The instrumentation of mke2fs shows that the first fsync() on /dev/loop0 (in lib/ext2fs/closefs.c) which is failing with EIO.

I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using dm_error, or some such), and somehow the errseq_t for the loop device isn't getting reset, or the inode for the underlying backing file, had an unitialized errseq_t.

Quoted 6 lines

Can you take a closer look at this?

Thanks,

- Ted

Thanks Ted. I'm not that familiar with the loopdev code, but after giving it a quick look, I suspect that you're correct. We probably need to do something like reset the loop device's bd_inode->i_mapping->wb_err back to zero when we detach the file that backs it.

I wonder if we could roll a test that would do:

create a scratch fs on a dm-error dev with a file on it set up a loop device on that file
have the backing device of the scratch file throw errors write to the device
detach loop device
clear dm-error condition
delete file and recreate it
attach same loop device to new file
fsync loop device

My suspicion is that that last fsync would throw an error now and it wouldn't have before.

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 54 lines · Jeff Layton
Quoted 29 lines · Theodore Y. Ts'o

Hi Matthew,

Commit b4678df184b: "errseq: Always report a writeback error once" appears to be causing xfstests regressions. For ext4, running "gce-xfstests -c 4k -g auto" will result in reliable shared/298 failures which go away if I revert b4678df184b.

Darrick has also reported occasional generic/047 failures, which I have seen at least once as well. I believe two are linked, because after instrumenting mke2fs in shared/298, the failure is happening after creating a new 300 MB file:

dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null

creating a new loop device

loop_dev=$(_create_loop_device $img_file)

... and then run mke2fs on that loop device.

The instrumentation of mke2fs shows that the first fsync() on /dev/loop0 (in lib/ext2fs/closefs.c) which is failing with EIO.

I haven't had a chance to really drill down on it, but I think what is going on is there is some former test which exercises an error path (using dm_error, or some such), and somehow the errseq_t for the loop device isn't getting reset, or the inode for the underlying backing file, had an unitialized errseq_t.

Quoted 6 lines

Can you take a closer look at this?

Thanks,

- Ted

Thanks Ted. I'm not that familiar with the loopdev code, but after giving it a quick look, I suspect that you're correct. We probably need to do something like reset the loop device's bd_inode->i_mapping->wb_err back to zero when we detach the file that backs it.

I wonder if we could roll a test that would do:

create a scratch fs on a dm-error dev with a file on it set up a loop device on that file
have the backing device of the scratch file throw errors write to the device
detach loop device
clear dm-error condition
delete file and recreate it
attach same loop device to new file
fsync loop device

Quoted 2 lines

My suspicion is that that last fsync would throw an error now and it wouldn't have before.

I /think/ it's because inode_init_always doesn't clear mapping->wb_err (even though it clears mapping->flags) when recycling struct inodes. Will send patch shortly.

--D

Quoted 2 lines
Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw

PATCH 1/1 fs: clear writeback errors in inode_init_always

In reply to Jeff Layton

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    2 ++
 2 files changed, 3 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +2 -0
index 164350d91efc..f4d672cb60a4 100644
@@ -298,6 +298,7 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +307,7 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
 	return error;
 }
 
#5 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Quoted 11 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Quoted 4 lines

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

Quoted 17 lines

+++ b/fs/xfs/xfs_icache.c

fs/xfs/xfs_icache.c +2 -0
@@ -298,6 +298,7 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +307,7 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
 	return error;
 }
 
#11 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong
Quoted 15 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Quoted 6 lines

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

This may fix the generic/047 failure, but alas, it does not address the shared/298 failure.

Jeff's theory that we may need to clear the errseq_t information after detaching the loop device makes sense to me; and in the case of the loop device, we wouldn't be initializing the inode, so your patch wouldn't do anything about that particular case.

- Ted

#12 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 23 lines · Theodore Y. Ts'o
Quoted 15 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Quoted 6 lines

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Quoted 9 lines

This may fix the generic/047 failure, but alas, it does not address the shared/298 failure.

Jeff's theory that we may need to clear the errseq_t information after detaching the loop device makes sense to me; and in the case of the loop device, we wouldn't be initializing the inode, so your patch wouldn't do anything about that particular case.

I poked at this a bit this morning and found that if I ran generic/361 twice in a row that I'd see a failure similar to what you were seeing. This patch seems to fix it for me.

Ted, could you test this against your reproducer? If this works for you I'll plan to flesh out the patch description and get this into -next and eventually to Linus before the next release.

Thanks,
Jeff

-------------------------8<--------------------

[PATCH] loop: clear wb_err in bd_inode when detaching backing file

---
 drivers/block/loop.c | 1 +
 1 file changed, 1 insertion(+)
drivers/block/loop.c +1 -0
index 5d4e31655d96..55cf554bc914 100644
@@ -1068,6 +1068,7 @@ static int loop_clr_fd(struct loop_device *lo)
 	if (bdev) {
 		bdput(bdev);
 		invalidate_bdev(bdev);
+		bdev->bd_inode->i_mapping->wb_err = 0;
 	}
 	set_capacity(lo->lo_disk, 0);
 	loop_sysfs_exit(lo);
Signature · 1 line
2.17.0
#13 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 1 line · Jeff Layton

[PATCH] loop: clear wb_err in bd_inode when detaching backing file

Is this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.

I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.

#14 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton

[PATCH] loop: clear wb_err in bd_inode when detaching backing file

Is this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.

I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.

This is a bit of a grey area, for sure. FWIW, I looked at doing this in invalidate_bdev, but backed off since I wasn't sure about the other callers.

I still think it probably is the right thing to do. I'd think that detaching the backing store is sort of an indicator that you really don't care about it anymore (including any stored errors concerning it).

OTOH, my track record on predicting what applications care about is pretty abysmal. I'll go with whatever the consensus is here...

Signature · 1 line
Jeff Layton <[email protected]>
#15 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Matthew Wilcox
Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton

[PATCH] loop: clear wb_err in bd_inode when detaching backing file

Is this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.

I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.

You can't detach the backing store if there are any open file descriptors (or if there is another loop device keeping the loop device open, or if there is a file system mounted on it, etc.).

If you can detach the backing store, once you detach the backing store, the loop device is *gone*. The user of /dev/loop0 will very likely be a completely different backing store, so returning an error simply doesn't make any sense. There is a very good chance it will be a completely different backing store, with potentially a different user, so returning a carried over error is just going confuse, annoy, and anger the user.....

- Ted

#20 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Jeff Layton
Quoted 8 lines · Jeff Layton

I poked at this a bit this morning and found that if I ran generic/361 twice in a row that I'd see a failure similar to what you were seeing. This patch seems to fix it for me.

Ted, could you test this against your reproducer? If this works for you I'll plan to flesh out the patch description and get this into -next and eventually to Linus before the next release.

Yup, this fixes shared/298 for me. Many thanks!!

- Ted

#16 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Theodore Y. Ts'o
Quoted 17 lines · Theodore Y. Ts'o
Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Jeff Layton

[PATCH] loop: clear wb_err in bd_inode when detaching backing file

Is this the right thing to do? Putting the test-suite aside for the moment, if I have a loop device on a file and I hit a real error on the storage backing that file, I don't see why detaching the loop device from the file should clear the error.

I'm really tempted to say that we should fix the test-suite to consume the error; once it's been read by at least one reader, it won't go back to zero, but neither will it show up for new readers.

You can't detach the backing store if there are any open file descriptors (or if there is another loop device keeping the loop device open, or if there is a file system mounted on it, etc.).

Quoted 7 lines

If you can detach the backing store, once you detach the backing store, the loop device is *gone*. The user of /dev/loop0 will very likely be a completely different backing store, so returning an error simply doesn't make any sense. There is a very good chance it will be a completely different backing store, with potentially a different user, so returning a carried over error is just going confuse, annoy, and anger the user.....

Oh! I misunderstood. I thought that bd_inode->i_mapping pointed to lo_backing_file->f_mapping.

So how about this to preserve the error _in the file with the error_?

	if (bdev) {
		bdput(bdev);
		invalidate_bdev(bdev);
+		filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err;
+		bdev->bd_inode->i_mapping->wb_err = 0;
	}
	set_capacity(lo->lo_disk, 0);
	loop_sysfs_exit(lo);
#17 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 10 lines · Matthew Wilcox

Oh! I misunderstood. I thought that bd_inode->i_mapping pointed to lo_backing_file->f_mapping.

So how about this to preserve the error _in the file with the error_?

	if (bdev) {
		bdput(bdev);
		invalidate_bdev(bdev);
(quoted diff) +2 -0
+		filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err;
+		bdev->bd_inode->i_mapping->wb_err = 0;
Quoted 3 lines
	}
	set_capacity(lo->lo_disk, 0);
	loop_sysfs_exit(lo);

I don't think it's necessary. wb_err on the underlying file would have already been set when the error was set. So if someone tried calling fsync(2) on the underlying file, it would have collected the error already. Re-setting the error when we detach the loop device doesn't seem to make any sense.

- Ted

#18 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 6 lines · Theodore Y. Ts'o
Quoted 5 lines · Matthew Wilcox

Oh! I misunderstood. I thought that bd_inode->i_mapping pointed to lo_backing_file->f_mapping.

So how about this to preserve the error _in the file with the error_?

Quoted 15 lines
Quoted 8 lines
	if (bdev) {
		bdput(bdev);
		invalidate_bdev(bdev);
(quoted diff) +2 -0
+		filp->f_mapping->wb_err = bdev->bd_inode->i_mapping->wb_err;
+		bdev->bd_inode->i_mapping->wb_err = 0;
	}
	set_capacity(lo->lo_disk, 0);
	loop_sysfs_exit(lo);

I don't think it's necessary. wb_err on the underlying file would have already been set when the error was set. So if someone tried calling fsync(2) on the underlying file, it would have collected the error already. Re-setting the error when we detach the loop device doesn't seem to make any sense.

(cc'ing linux-block)

Yes, the error would have been set on the original file already. As long as lo_req_flush or __loop_update_dio weren't called before we detach the device, it should still be possible to call fsync on the original fd and get back the error.

As a side note, it looks like __loop_update_dio will discard an error from vfs_fsync, so certain ioctls against a loop device can cause errors to be lost. It seems like those ought to get propagated to userland or to the blockdev's mapping somehow.

Signature · 1 line
Jeff Layton <[email protected]>
#19 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 5 lines · Jeff Layton

As a side note, it looks like __loop_update_dio will discard an error from vfs_fsync, so certain ioctls against a loop device can cause errors to be lost. It seems like those ought to get propagated to userland or to the blockdev's mapping somehow.

I'm not sure it's worth it. All of the ioctls that call loop_update_dio are used just to configure the loop device. In practice they are never called once the loop device is actually running. It might be a better choice is to just define that errors get reset if there is any attempt to reconfigure the loop device.

One of these ioctls change the offset that a loop device maps onto the backing file. So for example, while offset 0 of the loop device might have corresponds beginning of the backing file, after executing the ioctl, offset 0 of the loop device no corresponds to offset 2MB of the backing store. This might happen if we are resetting the loop device to point at the first partition of a disk image, for example. I really don't think that preserving the error status when we are doing that kind of radical configuration makes sense.

Or for example, when we change the block size of the loop device; if the underlying backing store is an IBM Mainframe DASD with a 2k block size, the reason why the error was signalled was because there was an attempt to write a 1k block onto a 2k block device. So again, resetting the error status of the loop device is the right thing to do.

The final thing that's worth perhaps exploring is whether or not we need all of these knobs and ioctls. In particular, LOOP_CHANGE_FD is used by losetup. It exists because back in prehistory, some distribution installer needed it for some obscure corner case. If I remember correctly, it had to do with switching out the initramfs floppy disk with the root file system floppy disk. We've wasted developer bandwidth trying to deal with syzbot complaints about that ioctl, and it's not clear it was worth the effort, because it's not clear any real code actually _needs_ that ioctl. It might have been easier to comment out the ioctl, and see if anyone screamed.

Given that loop device is maintianed mostly on the margins, and it's not clear that lots of complicated error handling during setup is really necessary, this is basically a please to keep things simple, or at least no more complicated than it has to be.

Cheers,

- Ted

#6 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Matthew Wilcox
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Fixed, thanks.

Quoted 11 lines
Quoted 11 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Quoted 7 lines
Quoted 4 lines

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.

--D

Quoted 21 lines
Quoted 10 lines

+++ b/fs/xfs/xfs_icache.c

fs/xfs/xfs_icache.c +1 -0
@@ -298,6 +298,7 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
 
 	error = inode_init_always(mp->m_super, inode);
 

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 313

Quoted 7 lines
fs/xfs/xfs_icache.c +1 -0
@@ -306,6 +307,7 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
 	return error;
 }
 

--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

PATCH v2 1/1 fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 2 files changed, 10 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +9 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);
 	return error;
 }
 
Reviewed-by on v2 1/1
#22 lore raw

Re: [PATCH v2] fs: clear writeback errors in inode_init_always

Quoted 57 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 2 files changed, 10 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +4 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 320

Quoted 10 lines
fs/xfs/xfs_icache.c +5 -0
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);
 	return error;
 }
 
#7 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Fixed, thanks.

Quoted 16 lines
Quoted 15 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Quoted 9 lines
Quoted 2 lines

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.

Yes, I think the patch you sent makes sense.

Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).

We could (in principle) pass a pointer to file->f_wb_err to most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.

Quoted 22 lines
Quoted 22 lines
Quoted 10 lines

+++ b/fs/xfs/xfs_icache.c

fs/xfs/xfs_icache.c +1 -0
@@ -298,6 +298,7 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
 
 	error = inode_init_always(mp->m_super, inode);
 

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 313

Quoted 7 lines
fs/xfs/xfs_icache.c +1 -0
@@ -306,6 +307,7 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
 	return error;
 }
 

--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
Jeff Layton <[email protected]>
#23 lore raw

Re: [PATCH v2] fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong
Quoted 28 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 2 files changed, 10 insertions(+)

...

Quoted 16 lines
fs/xfs/xfs_icache.c +4 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317

Quoted 8 lines
fs/xfs/xfs_icache.c +5 -0
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);

I'm wondering how safe this is. Can't the associated on-disk inode have been unlinked and reallocated anew across this kind of reinit of the in-core ip?

Brian

Quoted 7 lines
fs/xfs/xfs_icache.c +0 -1
 	return error;
 }
 
--

To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw

Re: [PATCH v2] fs: clear writeback errors in inode_init_always

Quoted 45 lines · Brian Foster
Quoted 28 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 2 files changed, 10 insertions(+)

...

Quoted 15 lines
fs/xfs/xfs_icache.c +4 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
Quoted 13 lines

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317

Quoted 9 lines
fs/xfs/xfs_icache.c +5 -0
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);

I'm wondering how safe this is. Can't the associated on-disk inode have been unlinked and reallocated anew across this kind of reinit of the in-core ip?

Oops, yeah, xfs_ialloc ought to clear those error states unconditionally when allocating a new on-disk inode.

--D

Quoted 13 lines

Brian

Quoted 7 lines
fs/xfs/xfs_icache.c +0 -1
 	return error;
 }
 
--

To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe fstests" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#25 lore raw

PATCH v3 1/1 fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 fs/xfs/xfs_inode.c  |    5 +++++
 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +9 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);
 	return error;
 }
 
fs/xfs/xfs_inode.c +5 -0
index 02eae5059231..6c47ea3e577b 100644
@@ -835,6 +835,11 @@ xfs_ialloc(
 			inode->i_mode |= S_ISGID;
 	}
 
+	/* Reset all vfs error state. */
+	inode->i_mapping->wb_err = 0;
+	clear_bit(AS_EIO, &inode->i_mapping->flags);
+	clear_bit(AS_ENOSPC, &inode->i_mapping->flags);
+
 	/*
 	 * If the group ID of the new file does not match the effective group
 	 * ID or one of the supplementary group IDs, the S_ISGID bit is cleared
Reviewed-by on v3 1/1
#26 lore raw

Re: [PATCH v3] fs: clear writeback errors in inode_init_always

Quoted 75 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 fs/xfs/xfs_inode.c  |    5 +++++
 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +9 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);
 	return error;
 }
 
fs/xfs/xfs_inode.c +0 -0
index 02eae5059231..6c47ea3e577b 100644

fs/xfs/xfs_inode.c xfs_ialloc( · line 838

Quoted 4 lines
fs/xfs/xfs_inode.c +1 -0
@@ -835,6 +835,11 @@ xfs_ialloc(
 			inode->i_mode |= S_ISGID;
 	}
 
+	/* Reset all vfs error state. */

I'd prefer a comment that reminds why we have to do this. E.g., something like:

/*
* In-core inode reinit carries over vfs error state. Reset it
* so we don't leak error state from a previous generation of
* the inode.
*/

... but feel free to reword that.

fs/xfs/xfs_inode.c xfs_ialloc( · line 842

Quoted 4 lines
fs/xfs/xfs_inode.c +4 -0
+	inode->i_mapping->wb_err = 0;
+	clear_bit(AS_EIO, &inode->i_mapping->flags);
+	clear_bit(AS_ENOSPC, &inode->i_mapping->flags);
+

I also wonder whether it might be cleaner to just reset this stuff in xfs_ifree() where we kill the i_mode and bump the gen and whatnot, but afaict this should work too. With the comment fix, at least:

Quoted 7 lines
fs/xfs/xfs_inode.c +0 -1
 	/*
 	 * If the group ID of the new file does not match the effective group
 	 * ID or one of the supplementary group IDs, the S_ISGID bit is cleared
--

To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#27 lore raw

Re: [PATCH v3] fs: clear writeback errors in inode_init_always

Quoted 84 lines · Brian Foster
Quoted 75 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 fs/xfs/xfs_inode.c  |    5 +++++
 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +9 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);
 	return error;
 }
 
fs/xfs/xfs_inode.c +0 -0
index 02eae5059231..6c47ea3e577b 100644

fs/xfs/xfs_inode.c xfs_ialloc( · line 838

Quoted 4 lines
fs/xfs/xfs_inode.c +1 -0
@@ -835,6 +835,11 @@ xfs_ialloc(
 			inode->i_mode |= S_ISGID;
 	}
 
+	/* Reset all vfs error state. */

I'd prefer a comment that reminds why we have to do this. E.g., something like:

Quoted 5 lines

/*
* In-core inode reinit carries over vfs error state. Reset it
* so we don't leak error state from a previous generation of
* the inode.
*/

I settled on:

/*
* In-core inode reinit does not clear vfs error state. Reset
* it so we don't leak error state from a previous generation of
* the inode.
*/

Quoted 12 lines

... but feel free to reword that.

fs/xfs/xfs_inode.c xfs_ialloc( · line 842

Quoted 4 lines
fs/xfs/xfs_inode.c +4 -0
+	inode->i_mapping->wb_err = 0;
+	clear_bit(AS_EIO, &inode->i_mapping->flags);
+	clear_bit(AS_ENOSPC, &inode->i_mapping->flags);
+

I also wonder whether it might be cleaner to just reset this stuff in xfs_ifree() where we kill the i_mode and bump the gen and whatnot, but afaict this should work too. With the comment fix, at least:

Quoted 1 line

Thanks for the review!

--D

Quoted 12 lines
Quoted 8 lines
fs/xfs/xfs_inode.c +0 -1
 	/*
 	 * If the group ID of the new file does not match the effective group
 	 * ID or one of the supplementary group IDs, the S_ISGID bit is cleared
--

To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#28 lore raw

Re: [PATCH v3] fs: clear writeback errors in inode_init_always

In reply to Darrick J. Wong
Quoted 59 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 fs/xfs/xfs_inode.c  |    5 +++++
 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +4 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317

Quoted 8 lines
fs/xfs/xfs_icache.c +5 -0
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);

I just don't see this as valid.

The inode has already been removed from the VFS cache when this happens (i.e. it's gone through iput_final()->evict->destroy_inode) and so any VFS-specific state is already invalid.

We're retaining XFS-specific inode information across the recycling of the xfs_inode (i.e. stuff that doesn't change *on-disk* across recycling an inode, but is reset by inode_init_always()). This is a creating a new life cycle for the VFS inode (i.e. I_NEW is set), so *there is no VFS state retained* across this operation from it's previous life.

Trying to do this is just going to lead us into nasty layering violations where the VFS state has to be hacked around just in case this recycling of the previous life cycle's state was the wrong thing to do.

fs/xfs/xfs_inode.c xfs_ialloc( · line 841

Quoted 12 lines
fs/xfs/xfs_inode.c +4 -0
index 02eae5059231..6c47ea3e577b 100644
@@ -835,6 +835,11 @@ xfs_ialloc(
 			inode->i_mode |= S_ISGID;
 	}
 
+	/* Reset all vfs error state. */
+	inode->i_mapping->wb_err = 0;
+	clear_bit(AS_EIO, &inode->i_mapping->flags);
+	clear_bit(AS_ENOSPC, &inode->i_mapping->flags);

Just like this. This is a sure sign we're doing the wrong thing in xfs_reinit_inode(). VFS inode lifecycle state is trashed at ->destroy_inode. It does not persist into new instantiations of VFS inode state, regardless of that new inode lifecycle gets to the I_NEW state...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#8 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Jeff Layton
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Fixed, thanks.

Quoted 16 lines
Quoted 15 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Quoted 9 lines
Quoted 2 lines

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.

Yes, I think the patch you sent makes sense.

Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).

Quoted 5 lines

We could (in principle) pass a pointer to file->f_wb_err to most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.

Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to fsync() failing and reporting the error to userspace?

This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on fsync(), but can be reported multiple times through different filesystem operations...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#29 lore raw

Re: [PATCH v3] fs: clear writeback errors in inode_init_always

In reply to Dave Chinner
Quoted 95 lines · Dave Chinner
Quoted 59 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSPC) bits. Unfortunately, we do not also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

---
v3: clear error state when allocating new inode
v2: retain AS_EIO/AS_ENOSPC across xfs inode reinit

---
 fs/inode.c          |    1 +
 fs/xfs/xfs_icache.c |    9 +++++++++
 fs/xfs/xfs_inode.c  |    5 +++++
 3 files changed, 15 insertions(+)
fs/inode.c +1 -0
index 13ceb98c3bd3..3b55391072f3 100644
@@ -178,6 +178,7 @@ int inode_init_always(struct super_block *sb, struct inode *inode)
 	mapping->a_ops = &empty_aops;
 	mapping->host = inode;
 	mapping->flags = 0;
+	mapping->wb_err = 0;
 	atomic_set(&mapping->i_mmap_writable, 0);
 	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
 	mapping->private_data = NULL;
fs/xfs/xfs_icache.c +4 -0
index 164350d91efc..d01f9544ff01 100644
@@ -298,6 +298,10 @@ xfs_reinit_inode(
 	uint64_t	version = inode_peek_iversion(inode);
 	umode_t		mode = inode->i_mode;
 	dev_t		dev = inode->i_rdev;
+	errseq_t	old_err = inode->i_mapping->wb_err;
+	bool		as_eio = test_bit(AS_EIO, &inode->i_mapping->flags);
+	bool		as_enospc = test_bit(AS_ENOSPC,
+					     &inode->i_mapping->flags);
 
 	error = inode_init_always(mp->m_super, inode);
 

fs/xfs/xfs_icache.c xfs_reinit_inode( · line 317

Quoted 8 lines
fs/xfs/xfs_icache.c +5 -0
@@ -306,6 +310,11 @@ xfs_reinit_inode(
 	inode_set_iversion_queried(inode, version);
 	inode->i_mode = mode;
 	inode->i_rdev = dev;
+	inode->i_mapping->wb_err = old_err;
+	if (as_eio)
+		set_bit(AS_EIO, &inode->i_mapping->flags);
+	if (as_enospc)
+		set_bit(AS_ENOSPC, &inode->i_mapping->flags);

I just don't see this as valid.

The inode has already been removed from the VFS cache when this happens (i.e. it's gone through iput_final()->evict->destroy_inode) and so any VFS-specific state is already invalid.

We're retaining XFS-specific inode information across the recycling of the xfs_inode (i.e. stuff that doesn't change *on-disk* across recycling an inode, but is reset by inode_init_always()). This is a creating a new life cycle for the VFS inode (i.e. I_NEW is set), so *there is no VFS state retained* across this operation from it's previous life.

Trying to do this is just going to lead us into nasty layering violations where the VFS state has to be hacked around just in case this recycling of the previous life cycle's state was the wrong thing to do.

Quoted 8 lines
fs/xfs/xfs_inode.c +0 -0
index 02eae5059231..6c47ea3e577b 100644
@@ -835,6 +835,11 @@ xfs_ialloc(
 			inode->i_mode |= S_ISGID;
 	}
 
Quoted 10 lines

fs/xfs/xfs_inode.c xfs_ialloc( · line 841

Quoted 4 lines
fs/xfs/xfs_inode.c +4 -0
+	/* Reset all vfs error state. */
+	inode->i_mapping->wb_err = 0;
+	clear_bit(AS_EIO, &inode->i_mapping->flags);
+	clear_bit(AS_ENOSPC, &inode->i_mapping->flags);

Just like this. This is a sure sign we're doing the wrong thing in xfs_reinit_inode(). VFS inode lifecycle state is trashed at ->destroy_inode. It does not persist into new instantiations of VFS inode state, regardless of that new inode lifecycle gets to the I_NEW state...

(Just reiterating a conversation we had on irc...)

So basically as far as the vfs is concerned the inode is gone and done and destroyed, and it has no expectation of state being preserved. Therefore, this patch can be trimmed to the inode_init_always part.

--D

Quoted 10 lines

Cheers,

Dave.

Signature · 6 lines
Dave Chinner
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

In reply to Dave Chinner
Quoted 51 lines · Dave Chinner
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Fixed, thanks.

Quoted 16 lines
Quoted 15 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Quoted 9 lines
Quoted 2 lines

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.

Yes, I think the patch you sent makes sense.

Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).

Quoted 5 lines

We could (in principle) pass a pointer to file->f_wb_err to most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.

Quoted 9 lines

Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to fsync() failing and reporting the error to userspace?

This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on fsync(), but can be reported multiple times through different filesystem operations...

If I get back an error on ioctl, why should I think that that has anything to do with writeback? For example, suppose I do:

write = 0
ioctl = -EIO
fsync = 0

The ioctl returns an error but the fsync returns 0. It would be reasonable to assume that my writes had made it to disk.

I think that we really do want to ensure that fsync always returns an error if writeback failed since the last fsync. That was sort of the point of the errseq_t work in the first place.

Signature · 1 line
Jeff Layton <[email protected]>
#10 lore raw

Re: [PATCH] fs: clear writeback errors in inode_init_always

Quoted 69 lines · Jeff Layton
Quoted 51 lines · Dave Chinner
Quoted 44 lines · Jeff Layton
Quoted 27 lines · Darrick J. Wong
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

In inode_init_always(), we clear the inode mapping flags, which clears any retained error (AS_EIO, AS_ENOSC) bits. Unfortunately, we do not

typo of ENOSPC in case you do a new version

Fixed, thanks.

Quoted 16 lines
Quoted 15 lines

also clear wb_err, which means that old mapping errors can leak through to new inodes.

This is crucial for the XFS inode allocation path because we recycle old in-core inodes and we do not want error state from an old file to leak into the new file. This bug was discovered by running generic/036 and generic/047 in a loop and noticing that the EIOs generated by the collision of direct and buffered writes in generic/036 would survive the remount between 036 and 047, and get reported to the fsyncs (on different files on a reformatted fs!) in generic/047.

Since we're changing the semantics of inode_init_always, we must also change xfs_reinit_inode to retain the writeback error state when we go to recover an inode that has been torn down in the vfs but not yet disposed of by XFS.

Quoted 9 lines
Quoted 2 lines

Don't you also need to preserve inode->i_mapping->flags across the reinitialisation for the users which aren't yet using ->wb_err?

At first I thought (mistakenly) that xfs had been completely converted, but digging further we still use the old filemap_check_errors. It seems reasonable to me that if we're going to resurrect an incore inode then we should try to hang on to AS_EIO/AS_ENOSPC for as long as we can.

Yes, I think the patch you sent makes sense.

Most of the XFS callers are using filemap_write_and_wait{_range}, and most of those seem to be called from ioctls (->setattr op being the notable exception).

Quoted 5 lines

We could (in principle) pass a pointer to file->f_wb_err to most of those callers, and use that to check for errors instead of looking for AS_EIO/AS_ENOSPC. We wouldn't want to advance the error cursor for those (as that should only be done in the context of an fsync), but it might be more reliable than using the flags here.

Quoted 9 lines

Why wouldn't we advance the error pointer? The data writeback error caused an operation to fail and the error has been reported to userspace, so how is that different to fsync() failing and reporting the error to userspace?

This seems like a slippery slope of inconsistency to me, where data writeback errors are only reported once on fsync(), but can be reported multiple times through different filesystem operations...

If I get back an error on ioctl, why should I think that that has anything to do with writeback? For example, suppose I do:

write = 0
ioctl = -EIO
fsync = 0

Quoted 6 lines

The ioctl returns an error but the fsync returns 0. It would be reasonable to assume that my writes had made it to disk.

I think that we really do want to ensure that fsync always returns an error if writeback failed since the last fsync. That was sort of the point of the errseq_t work in the first place.

This needs documentation explaining all these intricacies. Because I'm certain that we'll get it wrong again if the expected behaviour isn't all spelled out very clearly.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]