vfs: avoid sb->s_umount lock while changing bind-mount flags
RFC 1/1 vfs: avoid sb->s_umount lock while changing bind-mount flags
During remount of a bind mount (mount -o remount,bind,ro,... /mnt/mntpt), we currently take down_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we need s_umount lock only around the do_remount_sb() call. vfsmount_lock seems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside the down_write(&sb->s_umount).
I wanted to get comments if I am violating any assumption around this code. Another thing that I was curious about was if we need the {lock|unlock}_mount(path) around this code. Please advise.
--- fs/namespace.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
fs/namespace.c +4 -3
@@ -1838,20 +1838,21 @@ static int do_remount(struct path *path, int flags, int mnt_flags,if (err)return err; -down_write (&sb->s_umount);if (flags &MS_BIND ) err =change_mount_flags (path->mnt, flags);else if (!capable (CAP_SYS_ADMIN )) err = -EPERM ; -else +else { +down_write (&sb->s_umount); err =do_remount_sb (sb, flags, data,0 ); +up_write (&sb->s_umount); + }if (!err) {br_write_lock (&vfsmount_lock); mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; mnt->mnt.mnt_flags = mnt_flags;br_write_unlock (&vfsmount_lock); } -up_write (&sb->s_umount);if (!err) {br_write_lock (&vfsmount_lock);touch_mnt_namespace (mnt->mnt_ns);
Signature · 1 line
1.8.4
Al Viro
maintainerQuoted 10 lines · Aditya Kali
During remount of a bind mount (mount -o remount,bind,ro,...
/mnt/mntpt), we currently takedown_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we needs_umountlock only around thedo_remount_sb()call.vfsmount_lockseems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside thedown_write(&sb->s_umount).
What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt racing and ending up with that sucker rw and without nodev? As for lock_mount... nope - we carefully do *not* hold namespace_sem over any kind of fs operations. Anything getting stuck while holding it will have really nasty consequences.
So ->s_umount here is inelegant, but alternatives sucks worse...
Quoted 12 lines · Al Viro
Quoted 10 lines · Aditya Kali
During remount of a bind mount (mount -o remount,bind,ro,...
/mnt/mntpt), we currently takedown_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we needs_umountlock only around thedo_remount_sb()call.vfsmount_lockseems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside thedown_write(&sb->s_umount).
Quoted 2 lines
What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt racing and ending up with that sucker rw and without nodev?
Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the MS_BIND remount code outside of s_umount lock. Is it OK to not synchronize the non-MS_BIND do_remount_sb() call with change of mnt_flags in MS_BIND case?
--- fs/namespace.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)
fs/namespace.c +23 -14
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file) }EXPORT_SYMBOL (mnt_drop_write_file); +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static int mnt_make_readonly (struct mount *mnt) {int ret =0 ; -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags |=MNT_WRITE_HOLD ;/* * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)*/ smp_wmb (); mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD ; -br_write_unlock (&vfsmount_lock);return ret; } +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static void __mnt_unmake_readonly (struct mount *mnt) { -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags &= ~MNT_READONLY ; -br_write_unlock (&vfsmount_lock); }int sb_prepare_remount_readonly (struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags, int mnt_flags,if (err)return err; -down_write (&sb->s_umount); -if (flags &MS_BIND ) +if (flags &MS_BIND ) { +br_write_lock (&vfsmount_lock); err =change_mount_flags (path->mnt, flags); -else if (!capable (CAP_SYS_ADMIN )) +if (!err) { + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; + } +br_write_unlock (&vfsmount_lock); + }else if (!capable (CAP_SYS_ADMIN )) err = -EPERM ; -else +else { +down_write (&sb->s_umount); err =do_remount_sb (sb, flags, data,0 ); -if (!err) { -br_write_lock (&vfsmount_lock); - mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; - mnt->mnt.mnt_flags = mnt_flags; -br_write_unlock (&vfsmount_lock); +if (!err) { +br_write_lock (&vfsmount_lock); + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; +br_write_unlock (&vfsmount_lock); + } +up_write (&sb->s_umount); } -up_write (&sb->s_umount);if (!err) {br_write_lock (&vfsmount_lock);touch_mnt_namespace (mnt->mnt_ns);
Signature · 1 line
1.8.4
Hi Al and other fs-developers,
Please let me know what you think about this patch.
Thanks,
Signature · 1 line
Aditya
Quoted 121 lines · Aditya Kali
Quoted 15 lines · Al Viro
Quoted 11 lines · Aditya Kali
During remount of a bind mount (mount -o remount,bind,ro,...
/mnt/mntpt), we currently takedown_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we needs_umountlock only around thedo_remount_sb()call.vfsmount_lockseems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside thedown_write(&sb->s_umount).Quoted 3 lines
What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
racing and ending up with that sucker rw and without nodev?Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the
MS_BINDremount code outside ofs_umountlock. Is it OK to not synchronize the non-MS_BINDdo_remount_sb()call with change ofmnt_flagsinMS_BINDcase?--- fs/namespace.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)fs/namespace.c +23
-15@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file) }EXPORT_SYMBOL (mnt_drop_write_file); +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static int mnt_make_readonly (struct mount *mnt) {int ret =0 ; -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags |=MNT_WRITE_HOLD ;/* * After storing MNT_WRITE_HOLD, we'll read the counters. This store @@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)*/ smp_wmb (); mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD ; -br_write_unlock (&vfsmount_lock);return ret; } +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static void __mnt_unmake_readonly (struct mount *mnt) { -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags &= ~MNT_READONLY ; -br_write_unlock (&vfsmount_lock); }int sb_prepare_remount_readonly (struct super_block *sb)@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags, int mnt_flags, if (err) return err; -down_write (&sb->s_umount); -if (flags &MS_BIND ) +if (flags &MS_BIND ) { +br_write_lock (&vfsmount_lock); err =change_mount_flags (path->mnt, flags); -else if (!capable (CAP_SYS_ADMIN )) +if (!err) { + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; + } +br_write_unlock (&vfsmount_lock); + }else if (!capable (CAP_SYS_ADMIN )) err = -EPERM ; - else +else { +down_write (&sb->s_umount); err =do_remount_sb (sb, flags, data,0 ); -if (!err) { -br_write_lock (&vfsmount_lock); - mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; - mnt->mnt.mnt_flags = mnt_flags; -br_write_unlock (&vfsmount_lock); +if (!err) { +br_write_lock (&vfsmount_lock); + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; +br_write_unlock (&vfsmount_lock); + } +up_write (&sb->s_umount); } -up_write (&sb->s_umount);if (!err) {br_write_lock (&vfsmount_lock);touch_mnt_namespace (mnt->mnt_ns); --1.8.4
+Ted Ts'o, Tejun Heo, Jens Axboe
Quoted 36 lines · Aditya Kali
Hi Al and other fs-developers,
Please let me know what you think about this patch.
Thanks,
Quoted 29 lines · Aditya Kali
Quoted 15 lines · Al Viro
Quoted 11 lines · Aditya Kali
During remount of a bind mount (mount -o remount,bind,ro,...
/mnt/mntpt), we currently takedown_write(&sb->s_umount). This causes the remount operation to get blocked behind writes occuring on device (possibly mounted somewhere else). We have observed that simply trying to change the bind-mount from read-write to read-only can take several seconds becuase writeback is in progress. Looking at the code it seems to me that we needs_umountlock only around thedo_remount_sb()call.vfsmount_lockseems enough to protect the flag change on the mount. So this patch fixes the locking so that changing of flags can happen outside thedown_write(&sb->s_umount).Quoted 3 lines
What's to prevent mount -o remount,ro /mnt and mount -o remount,rw,nodev /mnt
racing and ending up with that sucker rw and without nodev?Thanks for the reply! I see the problem in my patch. Please find the second attempt at this patch below. I have tried to keep the non-MS_BIND remount semantics same while moving the
MS_BINDremount code outside ofs_umountlock. Is it OK to not synchronize the non-MS_BINDdo_remount_sb()call with change ofmnt_flagsinMS_BINDcase?
--- fs/namespace.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-)
fs/namespace.c +23 -14
@@ -454,11 +454,13 @@ void mnt_drop_write_file(struct file *file) }EXPORT_SYMBOL (mnt_drop_write_file); +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static int mnt_make_readonly (struct mount *mnt) {int ret =0 ; -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags |=MNT_WRITE_HOLD ;/* * After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -492,15 +494,15 @@ static int mnt_make_readonly(struct mount *mnt)*/ smp_wmb (); mnt->mnt.mnt_flags &= ~MNT_WRITE_HOLD ; -br_write_unlock (&vfsmount_lock);return ret; } +/* +* Must be called under br_write_lock(&vfsmount_lock); +*/ static void __mnt_unmake_readonly (struct mount *mnt) { -br_write_lock (&vfsmount_lock); mnt->mnt.mnt_flags &= ~MNT_READONLY ; -br_write_unlock (&vfsmount_lock); }int sb_prepare_remount_readonly (struct super_block *sb)
@@ -1838,20 +1840,27 @@ static int do_remount(struct path *path, int flags, int mnt_flags,if (err)return err; -down_write (&sb->s_umount); -if (flags &MS_BIND ) +if (flags &MS_BIND ) { +br_write_lock (&vfsmount_lock); err =change_mount_flags (path->mnt, flags); -else if (!capable (CAP_SYS_ADMIN )) +if (!err) { + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; + } +br_write_unlock (&vfsmount_lock); + }else if (!capable (CAP_SYS_ADMIN )) err = -EPERM ; -else +else { +down_write (&sb->s_umount); err =do_remount_sb (sb, flags, data,0 ); -if (!err) { -br_write_lock (&vfsmount_lock); - mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; - mnt->mnt.mnt_flags = mnt_flags; -br_write_unlock (&vfsmount_lock); +if (!err) { +br_write_lock (&vfsmount_lock); + mnt_flags |= mnt->mnt.mnt_flags &MNT_PROPAGATION_MASK ; + mnt->mnt.mnt_flags = mnt_flags; +br_write_unlock (&vfsmount_lock); + } +up_write (&sb->s_umount); } -up_write (&sb->s_umount);if (!err) {br_write_lock (&vfsmount_lock);touch_mnt_namespace (mnt->mnt_ns);
Signature · 1 line
1.8.4
Al Viro
maintainerQuoted 7 lines · Aditya Kali
+Ted Ts'o, Tejun Heo, Jens Axboe
Quoted 3 lines · Aditya Kali
Hi Al and other fs-developers,
Please let me know what you think about this patch.
Don't top-post, please... What prevents a race between MS_BIND remounts and plain ones? Used to be serialized on ->s_umount, but...
Quoted 11 lines · Al Viro
Quoted 7 lines · Aditya Kali
+Ted Ts'o, Tejun Heo, Jens Axboe
Quoted 3 lines · Aditya Kali
Hi Al and other fs-developers,
Please let me know what you think about this patch.
Don't top-post, please... What prevents a race between
MS_BINDremounts and plain ones? Used to be serialized on ->s_umount, but...
The rational here is that MS_BIND remounts need to act on the bind mount-point only (struct vfsmount *) and not on the underlying superblock (struct super_block *). So, intuitively, it should not be necessary to take a superblock level lock for the MS_BIND case. As for the modification of mnt_flags on vfsmount, there is already vfsmount_lock taken in both MS_BIND and non-MS_BIND case to prevent the race.
Following is the example that demonstrates the problem that I am trying to address with this patch:
(1) /dev/sda is mounted at /mnt/sda
(2) /mnt/sda/users/user1 is bind mounted at /home/user1/ ; /mnt/sda/users/user2 is bind mounted at /home/user2/
(3) user1 is doing buffered writes in /home/user1/logs/
(4) admin tries to make /home/user2/bin/ read-only using a bind-mounts:
$ mount --bind /home/user2/bin/ /home/user2/bin/ # this is fast
$ mount --bind -o remount,ro /home/user2/bin/ # this blocks
behind any writeback happening on sda (because of sb->s_umount write
lock)Please advise.
Thanks,
Signature · 1 line
Aditya