Skip to contentSkip to conversation

vfs: avoid sb->s_umount lock while changing bind-mount flags

RFC In review Aditya Kali 2 participants 7 messages Sep 16, 2013 – Sep 30, 2013 VFS

lore
Share & tools
lore
#0 lore raw

RFC 1/1 vfs: avoid sb->s_umount lock while changing bind-mount flags

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).

I wanted to get comments if I am violating any assumption around this code. Another thing that I was curious about was if we need the {lock|unlock}_mount(path) around this code. Please advise.

---
 fs/namespace.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
fs/namespace.c +4 -3
index da5c494..4b9c839 100644
@@ -1838,20 +1838,21 @@ static int do_remount(struct path *path, int flags, int mnt_flags,
 	if (err)
 		return err;
 
-	down_write(&sb->s_umount);
 	if (flags & MS_BIND)
 		err = change_mount_flags(path->mnt, flags);
 	else if (!capable(CAP_SYS_ADMIN))
 		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
 		err = do_remount_sb(sb, flags, data, 0);
+		up_write(&sb->s_umount);
+	}
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
 		mnt->mnt.mnt_flags = mnt_flags;
 		br_write_unlock(&vfsmount_lock);
 	}
-	up_write(&sb->s_umount);
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		touch_mnt_namespace(mnt->mnt_ns);
Signature · 1 line
1.8.4
maintainer
#1 lore raw
Quoted 10 lines · Aditya Kali

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).

What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt racing and ending up with that sucker rw and without nodev? As for lock_mount... nope - we carefully do *not* hold namespace_sem over any kind of fs operations. Anything getting stuck while holding it will have really nasty consequences.

So ->s_umount here is inelegant, but alternatives sucks worse...

#2 lore raw
Quoted 12 lines · Al Viro
Quoted 10 lines · Aditya Kali

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).

Quoted 2 lines

What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt racing and ending up with that sucker rw and without nodev?

Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the MS_BIND remount code outside of s_umount lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() call with change of mnt_flags in MS_BIND case?

---
 fs/namespace.c | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)
fs/namespace.c +23 -14
index da5c494..25c4faf 100644
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
 }
 EXPORT_SYMBOL(mnt_drop_write_file);
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static int mnt_make_readonly(struct mount *mnt)
 {
 	int ret = 0;
 
-	br_write_lock(&vfsmount_lock);
 	mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
 	/*
 	 * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
 	 */
 	smp_wmb();
 	mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
-	br_write_unlock(&vfsmount_lock);
 	return ret;
 }
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static void __mnt_unmake_readonly(struct mount *mnt)
 {
-	br_write_lock(&vfsmount_lock);
 	mnt->mnt.mnt_flags &= ~MNT_READONLY;
-	br_write_unlock(&vfsmount_lock);
 }
 
 int sb_prepare_remount_readonly(struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags, int mnt_flags,
 	if (err)
 		return err;
 
-	down_write(&sb->s_umount);
-	if (flags & MS_BIND)
+	if (flags & MS_BIND) {
+		br_write_lock(&vfsmount_lock);
 		err = change_mount_flags(path->mnt, flags);
-	else if (!capable(CAP_SYS_ADMIN))
+		if (!err) {
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+		}
+		br_write_unlock(&vfsmount_lock);
+	} else if (!capable(CAP_SYS_ADMIN))
 		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
 		err = do_remount_sb(sb, flags, data, 0);
-	if (!err) {
-		br_write_lock(&vfsmount_lock);
-		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
-		mnt->mnt.mnt_flags = mnt_flags;
-		br_write_unlock(&vfsmount_lock);
+		if (!err) {
+			br_write_lock(&vfsmount_lock);
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+			br_write_unlock(&vfsmount_lock);
+		}
+		up_write(&sb->s_umount);
 	}
-	up_write(&sb->s_umount);
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		touch_mnt_namespace(mnt->mnt_ns);
Signature · 1 line
1.8.4
#3 lore raw

Hi Al and other fs-developers,

Please let me know what you think about this patch.

Thanks,

Signature · 1 line
Aditya
Quoted 121 lines · Aditya Kali
Quoted 15 lines · Al Viro
Quoted 11 lines · Aditya Kali

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).

Quoted 3 lines

What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
racing and ending up with that sucker rw and without nodev?

Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the MS_BIND remount code outside of s_umount lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() call with change of mnt_flags in MS_BIND case?

---
 fs/namespace.c | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)
fs/namespace.c +23 -15
index da5c494..25c4faf 100644
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
 }
 EXPORT_SYMBOL(mnt_drop_write_file);
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static int mnt_make_readonly(struct mount *mnt)
 {
        int ret = 0;
 
-       br_write_lock(&vfsmount_lock);
        mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
        /*
         * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
         */
        smp_wmb();
        mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
-       br_write_unlock(&vfsmount_lock);
        return ret;
 }
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static void __mnt_unmake_readonly(struct mount *mnt)
 {
-       br_write_lock(&vfsmount_lock);
        mnt->mnt.mnt_flags &= ~MNT_READONLY;
-       br_write_unlock(&vfsmount_lock);
 }
 
 int sb_prepare_remount_readonly(struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags,
 int mnt_flags,
 
        if (err)
                return err;
 
-       down_write(&sb->s_umount);
-       if (flags & MS_BIND)
+       if (flags & MS_BIND) {
+               br_write_lock(&vfsmount_lock);
                err = change_mount_flags(path->mnt, flags);
-       else if (!capable(CAP_SYS_ADMIN))
+               if (!err) {
+                       mnt_flags |= mnt->mnt.mnt_flags &
 MNT_PROPAGATION_MASK;
+                       mnt->mnt.mnt_flags = mnt_flags;
+               }
+               br_write_unlock(&vfsmount_lock);
+       } else if (!capable(CAP_SYS_ADMIN))
 
                err = -EPERM;
-       else
+       else {
+               down_write(&sb->s_umount);
                err = do_remount_sb(sb, flags, data, 0);
-       if (!err) {
-               br_write_lock(&vfsmount_lock);
-               mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
-               mnt->mnt.mnt_flags = mnt_flags;
-               br_write_unlock(&vfsmount_lock);
+               if (!err) {
+                       br_write_lock(&vfsmount_lock);
+                       mnt_flags |= mnt->mnt.mnt_flags &
 MNT_PROPAGATION_MASK;
+                       mnt->mnt.mnt_flags = mnt_flags;
+                       br_write_unlock(&vfsmount_lock);
+               }
+               up_write(&sb->s_umount);
        }
-       up_write(&sb->s_umount);
        if (!err) {
                br_write_lock(&vfsmount_lock);
                touch_mnt_namespace(mnt->mnt_ns);
--

1.8.4

#4 lore raw

+Ted Ts'o, Tejun Heo, Jens Axboe

Quoted 36 lines · Aditya Kali

Hi Al and other fs-developers,

Please let me know what you think about this patch.

Thanks,

Quoted 29 lines · Aditya Kali
Quoted 15 lines · Al Viro
Quoted 11 lines · Aditya Kali

During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).

Quoted 3 lines

What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
racing and ending up with that sucker rw and without nodev?

Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the MS_BIND remount code outside of s_umount lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() call with change of mnt_flags in MS_BIND case?

---
 fs/namespace.c | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)
fs/namespace.c +23 -14
index da5c494..25c4faf 100644
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file)
 }
 EXPORT_SYMBOL(mnt_drop_write_file);
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static int mnt_make_readonly(struct mount *mnt)
 {
 	int ret = 0;
 
-	br_write_lock(&vfsmount_lock);
 	mnt->mnt.mnt_flags |= MNT_WRITE_HOLD;
 	/*
 	 * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)
 	 */
 	smp_wmb();
 	mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD;
-	br_write_unlock(&vfsmount_lock);
 	return ret;
 }
 
+/*
+ * Must be called under br_write_lock(&vfsmount_lock);
+ */
 static void __mnt_unmake_readonly(struct mount *mnt)
 {
-	br_write_lock(&vfsmount_lock);
 	mnt->mnt.mnt_flags &= ~MNT_READONLY;
-	br_write_unlock(&vfsmount_lock);
 }
 
 int sb_prepare_remount_readonly(struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags, int mnt_flags,
 	if (err)
 		return err;
 
-	down_write(&sb->s_umount);
-	if (flags & MS_BIND)
+	if (flags & MS_BIND) {
+		br_write_lock(&vfsmount_lock);
 		err = change_mount_flags(path->mnt, flags);
-	else if (!capable(CAP_SYS_ADMIN))
+		if (!err) {
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+		}
+		br_write_unlock(&vfsmount_lock);
+	} else if (!capable(CAP_SYS_ADMIN))
 		err = -EPERM;
-	else
+	else {
+		down_write(&sb->s_umount);
 		err = do_remount_sb(sb, flags, data, 0);
-	if (!err) {
-		br_write_lock(&vfsmount_lock);
-		mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
-		mnt->mnt.mnt_flags = mnt_flags;
-		br_write_unlock(&vfsmount_lock);
+		if (!err) {
+			br_write_lock(&vfsmount_lock);
+			mnt_flags |= mnt->mnt.mnt_flags & MNT_PROPAGATION_MASK;
+			mnt->mnt.mnt_flags = mnt_flags;
+			br_write_unlock(&vfsmount_lock);
+		}
+		up_write(&sb->s_umount);
 	}
-	up_write(&sb->s_umount);
 	if (!err) {
 		br_write_lock(&vfsmount_lock);
 		touch_mnt_namespace(mnt->mnt_ns);
Signature · 1 line
1.8.4
maintainer
#5 lore raw
Quoted 7 lines · Aditya Kali

+Ted Ts'o, Tejun Heo, Jens Axboe

Quoted 3 lines · Aditya Kali

Hi Al and other fs-developers,

Please let me know what you think about this patch.

Don't top-post, please... What prevents a race between MS_BIND remounts and plain ones? Used to be serialized on ->s_umount, but...

#6 lore raw
Quoted 11 lines · Al Viro
Quoted 7 lines · Aditya Kali

+Ted Ts'o, Tejun Heo, Jens Axboe

Quoted 3 lines · Aditya Kali

Hi Al and other fs-developers,

Please let me know what you think about this patch.

Don't top-post, please... What prevents a race between MS_BIND remounts and plain ones? Used to be serialized on ->s_umount, but...

The rational here is that MS_BIND remounts need to act on the bind mount-point only (struct vfsmount *) and not on the underlying superblock (struct super_block *). So, intuitively, it should not be necessary to take a superblock level lock for the MS_BIND case. As for the modification of mnt_flags on vfsmount, there is already vfsmount_lock taken in both MS_BIND and non-MS_BIND case to prevent the race.

Following is the example that demonstrates the problem that I am trying to address with this patch:

(1) /dev/sda is mounted at /mnt/sda
(2) /mnt/sda/users/user1 is bind mounted at /home/user1/ ; /mnt/sda/users/user2 is bind mounted at /home/user2/
(3) user1 is doing buffered writes in /home/user1/logs/
(4) admin tries to make /home/user2/bin/ read-only using a bind-mounts:

    $ mount --bind /home/user2/bin/ /home/user2/bin/    # this is fast
    $ mount --bind -o remount,ro /home/user2/bin/    # this blocks
behind any writeback happening on sda (because of sb->s_umount write
lock)

Please advise.

Thanks,

Signature · 1 line
Aditya