Skip to contentSkip to conversation

fs: Protect reconfiguration of sb read-write from racing writes

PATCH Applied Jan Kara 4 participants 8 messages Jun 15, 2023 – Jun 16, 2023 VFS

lore
Share & tools
lore

Jan Kara

author reviewer
#0 lore raw

PATCH 1/1 fs: Protect reconfiguration of sb read-write from racing writes

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

[1] https://lore.kernel.org/all/[email protected]/T/

---
 fs/super.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/super.c +10 -1
index 34afe411cf2b..6cd64961aa07 100644
@@ -903,6 +903,7 @@ int reconfigure_super(struct fs_context *fc)
 	struct super_block *sb = fc->root->d_sb;
 	int retval;
 	bool remount_ro = false;
+	bool remount_rw = false;
 	bool force = fc->sb_flags & SB_FORCE;
 
 	if (fc->sb_flags_mask & ~MS_RMT_MASK)
@@ -920,7 +921,7 @@ int reconfigure_super(struct fs_context *fc)
 		    bdev_read_only(sb->s_bdev))
 			return -EACCES;
 #endif
-
+		remount_rw = !(fc->sb_flags & SB_RDONLY) && sb_rdonly(sb);
 		remount_ro = (fc->sb_flags & SB_RDONLY) && !sb_rdonly(sb);
 	}
 
@@ -950,6 +951,14 @@ int reconfigure_super(struct fs_context *fc)
 			if (retval)
 				return retval;
 		}
+	} else if (remount_rw) {
+		/*
+		 * We set s_readonly_remount here to protect filesystem's
+		 * reconfigure code from writes from userspace until
+		 * reconfigure finishes.
+		 */
+		sb->s_readonly_remount = 1;
+		smp_wmb();
 	}
 
 	if (fc->ops->reconfigure) {
Signature · 1 line
2.35.3
#1 lore raw
Quoted 15 lines · Jan Kara

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

Quoted 3 lines

[1] https://lore.kernel.org/all/[email protected]/T/

---

So looking at the ext4 code this can only happen when you clear SB_RDONLY in ->reconfigure() too early (and the mount isn't MNT_READONLY). Afaict, this was fixed in:

a44be64bbecb ("ext4: don't clear SB_RDONLY when remounting r/w until quota is re-enabled")

by clearing SB_RDONLY late, right before returning from ->reconfigure() when everything's ready. So your change is not about fixing that bug in
[1] it's about making the vfs give the guarantee that an fs is free to clear SB_RDONLY because any ro<->rw transitions are protected via s_readonly_remount. Correct? It seems ok to me just making sure.

#2 lore raw
Quoted 5 lines · Christian Brauner

So looking at the ext4 code this can only happen when you clear SB_RDONLY in ->reconfigure() too early (and the mount isn't MNT_READONLY). Afaict, this was fixed in:

Quoted 7 lines

a44be64bbecb ("ext4: don't clear SB_RDONLY when remounting r/w until quota is re-enabled")

by clearing SB_RDONLY late, right before returning from ->reconfigure() when everything's ready. So your change is not about fixing that bug in
[1] it's about making the vfs give the guarantee that an fs is free to clear SB_RDONLY because any ro<->rw transitions are protected via s_readonly_remount. Correct? It seems ok to me just making sure.

Unfortunately we had to revert that commit because that broke r/o->r/w writes when quota was enabled. The problem is we need a way of enabling file system writes for internal purposes (e.g., because quota needs to set up quota inodes) but *not* allow userspace file system writes to occur until we are fully done with the remount process.

See the discussion here:

	https://lore.kernel.org/all/[email protected]/

The problem with the current state of the tree is commit dea9d8f7643f ("ext4: only check dquot_initialize_needed() when debugging") has caught real bugs in the past where the caller of ext4_xattr_block_set() failed to call dquot_initialize(inode). In addition, shutting up the warning doesn't fix the problem that while we hit this race where we have started remounting r/w, quota hasn't been initialized, quota tracking will get silently dropped, leading to the quota usage tracking no longer reflecting reality.

Jan's patch will fix this problem.

Cheers,

- Ted

author reviewer
#3 lore raw
Quoted 33 lines · Theodore Ts'o
Quoted 5 lines · Christian Brauner

So looking at the ext4 code this can only happen when you clear SB_RDONLY in ->reconfigure() too early (and the mount isn't MNT_READONLY). Afaict, this was fixed in:

Quoted 7 lines

a44be64bbecb ("ext4: don't clear SB_RDONLY when remounting r/w until quota is re-enabled")

by clearing SB_RDONLY late, right before returning from ->reconfigure() when everything's ready. So your change is not about fixing that bug in
[1] it's about making the vfs give the guarantee that an fs is free to clear SB_RDONLY because any ro<->rw transitions are protected via s_readonly_remount. Correct? It seems ok to me just making sure.

Unfortunately we had to revert that commit because that broke r/o->r/w writes when quota was enabled. The problem is we need a way of enabling file system writes for internal purposes (e.g., because quota needs to set up quota inodes) but *not* allow userspace file system writes to occur until we are fully done with the remount process.

See the discussion here:

	https://lore.kernel.org/all/[email protected]/

The problem with the current state of the tree is commit dea9d8f7643f ("ext4: only check dquot_initialize_needed() when debugging") has caught real bugs in the past where the caller of ext4_xattr_block_set() failed to call dquot_initialize(inode). In addition, shutting up the warning doesn't fix the problem that while we hit this race where we have started remounting r/w, quota hasn't been initialized, quota tracking will get silently dropped, leading to the quota usage tracking no longer reflecting reality.

Quoted 1 line

Jan's patch will fix this problem.

Yes, and to fully reveal the situation, we can indeed introduce ext4-private superblock state "only internal writes allowed" which can be used for quota setup. It just requires a bit of tidying and helper adjustment (in fact I have such patches on my private branch). But it has occured to me that generally it is easier if the filesystem's remount code doesn't have to care about racing writers on rw<->ro transitions since we have many filesystems and making sure all are getting this correct is ... tedious.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer Applied
#4 lore raw
In reply to Jan Kara
Quoted 15 lines · Jan Kara

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

Quoted 1 line

[...]

Applied to the vfs.misc branch of the vfs/vfs.git tree. Patches in the vfs.misc branch should appear in linux-next soon.

Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.

It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git
branch: vfs.misc

[1/1] fs: Protect reconfiguration of sb read-write from racing writes

      https://git.kernel.org/vfs/vfs/c/496de0b41695
#5 lore raw
In reply to Jan Kara
Quoted 44 lines · Jan Kara

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

[1] https://lore.kernel.org/all/[email protected]/T/

---
 fs/super.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
fs/super.c +2 -1
index 34afe411cf2b..6cd64961aa07 100644
@@ -903,6 +903,7 @@ int reconfigure_super(struct fs_context *fc)
 	struct super_block *sb = fc->root->d_sb;
 	int retval;
 	bool remount_ro = false;
+	bool remount_rw = false;
 	bool force = fc->sb_flags & SB_FORCE;
 
 	if (fc->sb_flags_mask & ~MS_RMT_MASK)
@@ -920,7 +921,7 @@ int reconfigure_super(struct fs_context *fc)
 		    bdev_read_only(sb->s_bdev))
 			return -EACCES;
 #endif
-
+		remount_rw = !(fc->sb_flags & SB_RDONLY) && sb_rdonly(sb);
 		remount_ro = (fc->sb_flags & SB_RDONLY) && !sb_rdonly(sb);
 	}
 
@@ -950,6 +951,14 @@ int reconfigure_super(struct fs_context *fc)
 			if (retval)

fs/super.c int reconfigure_super(struct fs_context *fc) · line 961

Quoted 10 lines
fs/super.c +8 -0
 				return retval;
 		}
+	} else if (remount_rw) {
+		/*
+		 * We set s_readonly_remount here to protect filesystem's
+		 * reconfigure code from writes from userspace until
+		 * reconfigure finishes.
+		 */
+		sb->s_readonly_remount = 1;
+		smp_wmb();

What does the magic random memory barrier do? What is it ordering, and what is it paired with?

This sort of thing is much better done with small helpers that encapsulate the necessary memory barriers:

sb_set_readonly_remount()
sb_clear_readonly_remount()

alongside the helper that provides the read-side check and memory barrier the write barrier is associated with.

I don't often ask for code to be cleaned up before a bug fix can be added, but I think this is one of the important cases where it does actually matter - we should never add undocumented memory barriers in the code like this...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author reviewer
#6 lore raw
Quoted 15 lines · Dave Chinner
Quoted 14 lines · Jan Kara

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

...

Quoted 12 lines

fs/super.c int reconfigure_super(struct fs_context *fc) · line 961

Quoted 8 lines
fs/super.c +8 -0
+	} else if (remount_rw) {
+		/*
+		 * We set s_readonly_remount here to protect filesystem's
+		 * reconfigure code from writes from userspace until
+		 * reconfigure finishes.
+		 */
+		sb->s_readonly_remount = 1;
+		smp_wmb();

What does the magic random memory barrier do? What is it ordering, and what is it paired with?

Quoted 8 lines

This sort of thing is much better done with small helpers that encapsulate the necessary memory barriers:

sb_set_readonly_remount()
sb_clear_readonly_remount()

alongside the helper that provides the read-side check and memory barrier the write barrier is associated with.

Fair remark. The new code including barrier just copies what happens a few lines above for remount read-only case (and what happens ib several other places throughout VFS code). I agree having helpers for this and actually documenting how the barriers are matching there is a good cleanup.

Quoted 4 lines

I don't often ask for code to be cleaned up before a bug fix can be added, but I think this is one of the important cases where it does actually matter - we should never add undocumented memory barriers in the code like this...

I've talked to Christian and we'll queue this as a separate cleanup. I'll post it shortly.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw
Quoted 35 lines · Jan Kara
Quoted 15 lines · Dave Chinner
Quoted 14 lines · Jan Kara

The reconfigure / remount code takes a lot of effort to protect filesystem's reconfiguration code from racing writes on remounting read-only. However during remounting read-only filesystem to read-write mode userspace writes can start immediately once we clear SB_RDONLY flag. This is inconvenient for example for ext4 because we need to do some writes to the filesystem (such as preparation of quota files) before we can take userspace writes so we are clearing SB_RDONLY flag before we are fully ready to accept userpace writes and syzbot has found a way to exploit this [1]. Also as far as I'm reading the code the filesystem remount code was protected from racing writes in the legacy mount path by the mount's MNT_READONLY flag so this is relatively new problem. It is actually fairly easy to protect remount read-write from racing writes using sb->s_readonly_remount flag so let's just do that instead of having to workaround these races in the filesystem code.

...

Quoted 18 lines

fs/super.c int reconfigure_super(struct fs_context *fc) · line 961

Quoted 8 lines
fs/super.c +8 -0
+	} else if (remount_rw) {
+		/*
+		 * We set s_readonly_remount here to protect filesystem's
+		 * reconfigure code from writes from userspace until
+		 * reconfigure finishes.
+		 */
+		sb->s_readonly_remount = 1;
+		smp_wmb();

What does the magic random memory barrier do? What is it ordering, and what is it paired with?

This sort of thing is much better done with small helpers that encapsulate the necessary memory barriers:

sb_set_readonly_remount()
sb_clear_readonly_remount()

Quoted 6 lines
Quoted 2 lines

alongside the helper that provides the read-side check and memory barrier the write barrier is associated with.

Fair remark. The new code including barrier just copies what happens a few lines above for remount read-only case (and what happens ib several other places throughout VFS code).

Yes, I saw that you'd copied that magic memory barrier. Good for consistency, but it doesn't answer any of the above questions either, so....

Quoted 10 lines

I agree having helpers for this and actually documenting how the barriers are matching there is a good cleanup.

Quoted 4 lines

I don't often ask for code to be cleaned up before a bug fix can be added, but I think this is one of the important cases where it does actually matter - we should never add undocumented memory barriers in the code like this...

I've talked to Christian and we'll queue this as a separate cleanup. I'll post it shortly.

Thanks!

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]