Skip to contentSkip to conversation

xfs: reset child dir '..' entry when unlinking child

PATCH In review Darrick J. Wong 4 participants 9 messages Jul 3, 2021 – Jul 7, 2021 XFS filesystem

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xfs: reset child dir '..' entry when unlinking child

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

To avoid this problem, reset B's dotdot entry to the root directory when unlinking directories, since the root directory cannot be removed.

---
 fs/xfs/xfs_inode.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)
fs/xfs/xfs_inode.c +13 -0
index 52be5c6d0b3b..03e25246e936 100644
@@ -2817,6 +2817,19 @@ xfs_remove(
 		error = xfs_droplink(tp, ip);
 		if (error)
 			goto out_trans_cancel;
+
+		/*
+		 * Point the unlinked child directory's ".." entry to the root
+		 * directory to eliminate back-references to inodes that may
+		 * get freed before the child directory is closed.  If the fs
+		 * gets shrunk, this can lead to dirent inode validation errors.
+		 */
+		if (dp->i_ino != tp->t_mountp->m_sb.sb_rootino) {
+			error = xfs_dir_replace(tp, ip, &xfs_name_dotdot,
+					tp->t_mountp->m_sb.sb_rootino, 0);
+			if (error)
+				return error;
+		}
 	} else {
 		/*
 		 * When removing a non-directory we need to log the parent
Reviewed-by on v1 1/1
#1 lore raw
Quoted 12 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

Quoted 4 lines

To avoid this problem, reset B's dotdot entry to the root directory when unlinking directories, since the root directory cannot be removed.

I think it is reasonable to clean dotdot for all deaddir cases (compared with skipping validating such dotdot dirent which sounds a bit hacky though since dotdot was actually outdated...)

Thanks,
Gao Xiang

Quoted 28 lines
---
 fs/xfs/xfs_inode.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)
fs/xfs/xfs_inode.c +13 -0
index 52be5c6d0b3b..03e25246e936 100644
@@ -2817,6 +2817,19 @@ xfs_remove(
 		error = xfs_droplink(tp, ip);
 		if (error)
 			goto out_trans_cancel;
+
+		/*
+		 * Point the unlinked child directory's ".." entry to the root
+		 * directory to eliminate back-references to inodes that may
+		 * get freed before the child directory is closed.  If the fs
+		 * gets shrunk, this can lead to dirent inode validation errors.
+		 */
+		if (dp->i_ino != tp->t_mountp->m_sb.sb_rootino) {
+			error = xfs_dir_replace(tp, ip, &xfs_name_dotdot,
+					tp->t_mountp->m_sb.sb_rootino, 0);
+			if (error)
+				return error;
+		}
 	} else {
 		/*
 		 * When removing a non-directory we need to log the parent
#2 lore raw
In reply to Darrick J. Wong
Quoted 12 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

Quoted 2 lines

To avoid this problem, reset B's dotdot entry to the root directory when unlinking directories, since the root directory cannot be removed.

Uggh. This causes extra overhead for every remove. Can't we make the verifieds deal with this situation instead of creating extra overhead? If we can't please at least limit it to file systems that do have parent pointers enabled.

#4 lore raw
In reply to Darrick J. Wong
Quoted 11 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

So we have a directory inode that is empty and unlinked but held open, with a back pointer to an invalid inode number? Which can never be followed, because the directory has been unlinked.

Can't this be handled in the inode verifier? This seems to me to be a pretty clear cut case where the ".." back pointer should always be considered invalid (because the parent dir has no existence guarantee once the child has been removed from it), not just in the situation where the filesystem has been shrunk...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#5 lore raw

Hi Dave and Christoph,

Quoted 17 lines · Dave Chinner
Quoted 11 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

So we have a directory inode that is empty and unlinked but held open, with a back pointer to an invalid inode number? Which can never be followed, because the directory has been unlinked.

Quoted 5 lines

Can't this be handled in the inode verifier? This seems to me to be a pretty clear cut case where the ".." back pointer should always be considered invalid (because the parent dir has no existence guarantee once the child has been removed from it), not just in the situation where the filesystem has been shrunk...

Yes, I agree all of this, this field can be handled by the inode verifier. The only concern I can think out might be fs freeze with a directory inode that is empty and unlinked but held open, and then recovery on a unpatched old kernels, not sure if such case can be handled properly by old kernel verifier.

Otherwise, it's also ok I think.

Thanks,
Gao Xiang

Quoted 7 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
In reply to Christoph Hellwig
Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

Quoted 4 lines
Quoted 2 lines

To avoid this problem, reset B's dotdot entry to the root directory when unlinking directories, since the root directory cannot be removed.

Uggh. This causes extra overhead for every remove.

Not that much overhead. A child directory can only be unlinked if it's empty; empty directories by definition contain only a dotdot entry, which meanns they're in shortform format; and we already have to log the inode to reflect the i_nlinks change.

Quoted 3 lines

Can't we make
the verifieds deal with this situation instead of creating extra overhead?

I'll address that in the other thread suggesting I "just fix the verifiers".

Quoted 2 lines

If we can't please at least limit it to file systems that do have parent pointers enabled.

Parent pointers haven't been merged yet; this is the '..' entry that has been stored in every directory since the beginning.

--D

#6 lore raw
In reply to Gao Xiang
Quoted 26 lines · Gao Xiang

Hi Dave and Christoph,

Quoted 17 lines · Dave Chinner
Quoted 11 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

So we have a directory inode that is empty and unlinked but held open, with a back pointer to an invalid inode number? Which can never be followed, because the directory has been unlinked.

Quoted 5 lines

Can't this be handled in the inode verifier? This seems to me to be a pretty clear cut case where the ".." back pointer should always be considered invalid (because the parent dir has no existence guarantee once the child has been removed from it), not just in the situation where the filesystem has been shrunk...

Quoted 5 lines

Yes, I agree all of this, this field can be handled by the inode verifier. The only concern I can think out might be fs freeze with a directory inode that is empty and unlinked but held open, and then recovery on a unpatched old kernels, not sure if such case can be handled properly by old kernel verifier.

I decided against changing the shortform directory verifier to ignore the dotdot pointer on nlink==0 directories because older kernels will still have the current behavior and that will cause recovery failure for the following sequence:

1. create A and B and delete A as per above, leave B open
2. shrink fs so that A is no longer a valid inode number
3. flush the shrink transaction to disk
4. futimens(B) (or anything to dirty the inode)
5. crash
6. boot old kernel
7. try to recover fs with old kernel, which will crash since B hadn't been inactivated

But I guess I'll have to go write an fstest to prove that I'm not making this up...

--D

Quoted 13 lines

Otherwise, it's also ok I think.

Thanks,
Gao Xiang

Quoted 7 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw
Quoted 38 lines · Darrick J. Wong
Quoted 26 lines · Gao Xiang

Hi Dave and Christoph,

Quoted 17 lines · Dave Chinner
Quoted 11 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

So we have a directory inode that is empty and unlinked but held open, with a back pointer to an invalid inode number? Which can never be followed, because the directory has been unlinked.

Quoted 5 lines

Can't this be handled in the inode verifier? This seems to me to be a pretty clear cut case where the ".." back pointer should always be considered invalid (because the parent dir has no existence guarantee once the child has been removed from it), not just in the situation where the filesystem has been shrunk...

Quoted 5 lines

Yes, I agree all of this, this field can be handled by the inode verifier. The only concern I can think out might be fs freeze with a directory inode that is empty and unlinked but held open, and then recovery on a unpatched old kernels, not sure if such case can be handled properly by old kernel verifier.

I decided against changing the shortform directory verifier to ignore the dotdot pointer on nlink==0 directories because older kernels will still have the current behavior and that will cause recovery failure for the following sequence:

Quoted 8 lines

1. create A and B and delete A as per above, leave B open
2. shrink fs so that A is no longer a valid inode number
3. flush the shrink transaction to disk
4. futimens(B) (or anything to dirty the inode)
5. crash
6. boot old kernel
7. try to recover fs with old kernel, which will crash since B hadn't been inactivated

Yup, I can see how that would happen, but patching the directory code still smells like band-aid to me on multiple levels.

At first glance, this seems like the first step in a on-going game of whack-a-mole. If shrink is changing how the on disk format needs to be evaluated in ways that older kernels can't safely deal with, then shrink needs to be setting an incompat feature bit to prevent older kernels from crashing trying to parse the on-disk format.

On deeper consideration after looking at the code, why is it even considered safe to be recovering pre- and post- shrink operations in a single log recovery operation? i.e. I see nothing in the shrink code that quiesces the log and flushes all the dirty metadata to stable storage prior to logging the superblock geometry changes and returning to userspace.

I can think of another situation where this might be problematic - recovering buffers containing unlinked inode lists updates. If the inode cluster buffers we recover from transactions before the shrink point to inodes that are beyond EOFS after the shrink, we recover them and write them to disk, only for them to be read again and written to disk when recovering subsequent transactions prior to shrink.

IOWs, we can have transient on-disk state during log recovery where stuff before the shrink transaction points to objects beyond the EOFS after the shrink, but are still considered to be valid by log recovery. If the device has already been shrunk, then log recovery has just created a transient corrupt on-disk state, so if log recovery fails at this point (for whatever reason), we've got a mess on our hands.

I suspect that this means we could also have problems in the AIL, too, but I haven't thought that far through this right now. The transient corrupt log recovery state can be fixed by forcing the AIL to be written back before we execute the shrink transaction so the shrink transaction ends up being written to the tail of the log. Doing this would also fix any sort of transient AIL ordering issue that might exist around a shrink operation....

So, yeah, I think this problem hints at bigger issues with shrink, on disk format interpretation and transient log recovery states. I think we need to finish off all the infrastructure shrink needs before we go any further with it....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#8 lore raw
Quoted 48 lines · Dave Chinner
Quoted 38 lines · Darrick J. Wong
Quoted 26 lines · Gao Xiang

Hi Dave and Christoph,

Quoted 17 lines · Dave Chinner
Quoted 11 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

While running xfs/168, I noticed a second source of post-shrink corruption errors causing shutdowns.

Let's say that directory B has a low inode number and is a child of directory A, which has a high number. If B is empty but open, and unlinked from A, B's dotdot link continues to point to A. If A is then unlinked and the filesystem shrunk so that A is no longer a valid inode, a subsequent AIL push of B will trip the inode verifiers because the dotdot entry points outside of the filesystem.

So we have a directory inode that is empty and unlinked but held open, with a back pointer to an invalid inode number? Which can never be followed, because the directory has been unlinked.

Quoted 5 lines

Can't this be handled in the inode verifier? This seems to me to be a pretty clear cut case where the ".." back pointer should always be considered invalid (because the parent dir has no existence guarantee once the child has been removed from it), not just in the situation where the filesystem has been shrunk...

Quoted 5 lines

Yes, I agree all of this, this field can be handled by the inode verifier. The only concern I can think out might be fs freeze with a directory inode that is empty and unlinked but held open, and then recovery on a unpatched old kernels, not sure if such case can be handled properly by old kernel verifier.

I decided against changing the shortform directory verifier to ignore the dotdot pointer on nlink==0 directories because older kernels will still have the current behavior and that will cause recovery failure for the following sequence:

Quoted 8 lines

1. create A and B and delete A as per above, leave B open
2. shrink fs so that A is no longer a valid inode number
3. flush the shrink transaction to disk
4. futimens(B) (or anything to dirty the inode)
5. crash
6. boot old kernel
7. try to recover fs with old kernel, which will crash since B hadn't been inactivated

Quoted 8 lines

Yup, I can see how that would happen, but patching the directory code still smells like band-aid to me on multiple levels.

At first glance, this seems like the first step in a on-going game of whack-a-mole. If shrink is changing how the on disk format needs to be evaluated in ways that older kernels can't safely deal with, then shrink needs to be setting an incompat feature bit to prevent older kernels from crashing trying to parse the on-disk format.

Not a quite excuse for myself, but it's not easy for me to realize there could be some problem with the outdated dotdot pointer pointing to already free blocks in advance since that's too detail without looking through the specific code considering my XFS knowledge, so I was always expecting for more reviews, and that's why this was marked as an experimental risky feature.

Quoted 16 lines

On deeper consideration after looking at the code, why is it even considered safe to be recovering pre- and post- shrink operations in a single log recovery operation? i.e. I see nothing in the shrink code that quiesces the log and flushes all the dirty metadata to stable storage prior to logging the superblock geometry changes and returning to userspace.

I can think of another situation where this might be problematic - recovering buffers containing unlinked inode lists updates. If the inode cluster buffers we recover from transactions before the shrink point to inodes that are beyond EOFS after the shrink, we recover them and write them to disk, only for them to be read again and written to disk when recovering subsequent transactions prior to shrink.

Quoted 7 lines

IOWs, we can have transient on-disk state during log recovery where stuff before the shrink transaction points to objects beyond the EOFS after the shrink, but are still considered to be valid by log recovery. If the device has already been shrunk, then log recovery has just created a transient corrupt on-disk state, so if log recovery fails at this point (for whatever reason), we've got a mess on our hands.

Yes, for such cases AIL needs to be pushed out before returning to the userspace to avoid device shrinking. Since before shrinking transaction the tail space was considered as valid. The shrinking transaction is already a sync transaction so it will trigger log force, and it needs a AIL push all as well before returning to the user space.

Quoted 9 lines

I suspect that this means we could also have problems in the AIL, too, but I haven't thought that far through this right now. The transient corrupt log recovery state can be fixed by forcing the AIL to be written back before we execute the shrink transaction so the shrink transaction ends up being written to the tail of the log. Doing this would also fix any sort of transient AIL ordering issue that might exist around a shrink operation....

Quoted 4 lines

So, yeah, I think this problem hints at bigger issues with shrink, on disk format interpretation and transient log recovery states. I think we need to finish off all the infrastructure shrink needs before we go any further with it....

Hmm.. That will avoid this indeed if more on-disk fields point to free blocks like this, honestly I didn't realize outdated dotdot pointer could point to already free blocks then a validator checks this in the beginning.

Thanks,
Gao Xiang

Quoted 7 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]