acct: fix possible deadlock in acct_pin_kill
PATCH v2 1/1 acct: fix possible deadlock in acct_pin_kill
This looks like an old bug, pre-dating the "Fixes" commit, but the "Fixes" commit did not handle it properly.
The bug recently surfaced as a lockdep possible deadlock warning with commit d1d04ef8572b ("ovl: stack file ops").
When acct_on() replaces one acct file with another, it takes sb_writers lock on new file sb and calls acct_pin_kill(old) before releasing the sb_writers lock.
If new file is on the same fs as old file, acct_pin_kill(old) fail to file_start_write_trylock() and skip writing the old file, because sb_writers (of new) is already taken by acct_on().
If new file is not on same fs as old file, this ordering violation creates an unneeded dependency between new sb_writers and old sb_writers, which may later be reported as possible deadlock.
This could result in an actual deadlock if acct file is replaced from an old file in overlayfs over "real fs" to a new file in "real fs". acct_on() takes freeze protection on "real fs" and tries to write to overlayfs file. overlayfs is not freeze protected so do_acct_process() can carry on with __kernel_write() to overlayfs file, which would try to take freeze protection on "real fs" and deadlock.
Reproducer of lockdep possible deadlock warning:
mkdir -p mnt upper lower work
touch upper/x upper/y
mount -t overlay none mnt -olowerdir=lower,upperdir=upper,workdir=work accton mnt/x
accton upper/y
======================================================
WARNING: possible circular locking dependency detected 4.19.0-rc1-xfstests #3424 Not tainted
------------------------------------------------------
accton/1390 is trying to acquire lock:
00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76
but task is already holding lock:
000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42
---
Hi Al,
I've pinged you several times about this patch.
Not sure if you had missed it or haven't had the time to look at it yet. syzbot has been complaining about the bug for a while now.
Fixes label claims to fix your commit, but I believe the bug was already there before your commit.
As you can see, I have a reproducer to demonstrate the manifestation of the bug in the case of switching acct file from overlayfs to real fs. This started to manifest with overlayfs stacked f_op.
I have made another claim about correctness of acct_on() when switching files on the same fs which seems obvious from the code, but did not bother to try to reproducer, because I doubt if anyone cares.
Thanks,
Amir.
Changes from v1:
- Add Reported-and-tested-by syzbot
kernel/acct.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/acct.c +1 -1
@@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)rcu_read_lock (); old =xchg (&ns->bacct, &acct->pin);mutex_unlock (&acct->lock); -pin_kill (old);mnt_drop_write (mnt); +pin_kill (old);mntput (mnt);return 0 ; }
Signature · 1 line
2.17.1
Al Viro
maintainerQuoted 18 lines · Amir Goldstein
This looks like an old bug, pre-dating the "Fixes" commit, but the "Fixes" commit did not handle it properly.
The bug recently surfaced as a lockdep possible deadlock warning with commit
d1d04ef8572b("ovl: stack file ops").When
acct_on()replaces one acct file with another, it takessb_writerslock on new file sb and callsacct_pin_kill(old)before releasing thesb_writerslock.If new file is on the same fs as old file,
acct_pin_kill(old)fail tofile_start_write_trylock()and skip writing the old file, becausesb_writers(of new) is already taken byacct_on().If new file is not on same fs as old file, this ordering violation creates an unneeded dependency between new
sb_writersand oldsb_writers, which may later be reported as possible deadlock.
Quoted 6 lines
This could result in an actual deadlock if acct file is replaced from an old file in overlayfs over "real fs" to a new file in "real fs".
acct_on()takes freeze protection on "real fs" and tries to write to overlayfs file. overlayfs is not freeze protected sodo_acct_process()can carry on with__kernel_write()to overlayfs file, which would try to take freeze protection on "real fs" and deadlock.
Huh? sb_writers is taken when we *open* the new file. Then we replace its ->path.mnt with a clone and transfer the write count from the original to new one. And close the old file while we are at it.
From sb_writers POV mainline has
sb_start_write(new_sb) // in open
sb_start_write(new_sb) // mnt_want_write() on clone
last write to old_sb, then sb_end_write(old_sb) // acct_pin_kill()
sb_end_write(new_sb) // mnt_drop_write(mnt)
and you flip the order of the last two lines.
Could you explain how exactly does your patch help whatever problem overlayfs has?
Al Viro
maintainerQuoted 5 lines · Al Viro
Huh?
sb_writersis taken when we *open* the new file. Then we replace its ->path.mnt with a clone and transfer the write count from the original to new one. And close the old file while we are at it.Quoted 1 line
From
sb_writersPOV mainline has
Quoted 8 lines
sb_start_write(new_sb)// in open
sb_start_write(new_sb)//mnt_want_write()on clone
last write toold_sb, thensb_end_write(old_sb)//acct_pin_kill()
sb_end_write(new_sb)//mnt_drop_write(mnt)
and you flip the order of the last two lines.Could you explain how exactly does your patch help whatever problem overlayfs has?
Ow... Sorry, -ENOCOFFEE...
Let me wake up properly and look at that thing again.
Al Viro
maintainerQuoted 13 lines · Al Viro
Quoted 11 lines · Al Viro
Huh?
sb_writersis taken when we *open* the new file. Then we replace its ->path.mnt with a clone and transfer the write count from the original to new one. And close the old file while we are at it.Quoted 1 line
From
sb_writersPOV mainline has
sb_start_write(new_sb)// in open
sb_start_write(new_sb)//mnt_want_write()on clone
last write toold_sb, thensb_end_write(old_sb)//acct_pin_kill()
sb_end_write(new_sb)//mnt_drop_write(mnt)
and you flip the order of the last two lines.
Quoted 6 lines
Quoted 2 lines
Could you explain how exactly does your patch help whatever problem overlayfs has?
Ow... Sorry,
-ENOCOFFEE...Let me wake up properly and look at that thing again.
OK, so... My first reaction had been complete BS. However, the same goes for your analysis - it's not an ordering problem at all. What happens is that we are replacing file->path.mnt with a clone and we want the write count contribution (file is opened for write) to be transferred. That's it. We do *NOT* want any kind of freeze protection for the duration of switchover.
IOW, the solution is to switch to __mnt_{want,drop}_write() for that switchover; we don't want to mess with freeze protection at all.
---
fs/internal.h +0 -2
@@ -89,9 +89,7 @@ extern int sb_prepare_remount_readonly(struct super_block *);extern void __init mnt_init (void ); -extern int __mnt_want_write (struct vfsmount *);extern int __mnt_want_write_file (struct file *); -extern void __mnt_drop_write (struct vfsmount *);extern void __mnt_drop_write_file (struct file *);/*
include/linux/mount.h +2 -0
@@ -87,6 +87,8 @@ extern bool mnt_may_suid(struct vfsmount *mnt);struct path ;extern struct vfsmount *clone_private_mount (const struct path *path); +extern int __mnt_want_write (struct vfsmount *); +extern void __mnt_drop_write (struct vfsmount *);struct file_system_type ;extern struct vfsmount *fc_mount (struct fs_context *fc);
kernel/acct.c +2 -2
@@ -227,7 +227,7 @@ static int acct_on(struct filename *pathname)filp_close (file,NULL );return PTR_ERR (internal); } - err =mnt_want_write (internal); + err =__mnt_want_write (internal);if (err) {mntput (internal);kfree (acct);
@@ -252,7 +252,7 @@ static int acct_on(struct filename *pathname) old =xchg (&ns->bacct, &acct->pin);mutex_unlock (&acct->lock);pin_kill (old); -mnt_drop_write (mnt); +__mnt_drop_write (mnt);mntput (mnt);return 0 ; }
Amir Goldstein
Tested-byQuoted 1 line · Al Viro
...
Quoted 11 lines
OK, so... My first reaction had been complete BS. However, the same goes for your analysis - it's not an ordering problem at all. What happens is that we are replacing
file->path.mntwith a clone and we want the write count contribution (file is opened for write) to be transferred. That's it. We do *NOT* want any kind of freeze protection for the duration of switchover.IOW, the solution is to switch to __mnt_{want,drop}_write() for that switchover; we don't want to mess with freeze protection at all.
Signed-off-by Al Viro <[email protected]>
Cool. That works for me.
Thanks for setting this straight.
You may add:
Thanks,
Amir.
Quoted 50 lines
---
fs/internal.h +0
-2@@ -89,9 +89,7 @@ extern int sb_prepare_remount_readonly(struct super_block *);extern void __init mnt_init (void ); -extern int __mnt_want_write (struct vfsmount *);extern int __mnt_want_write_file (struct file *); -extern void __mnt_drop_write (struct vfsmount *);extern void __mnt_drop_write_file (struct file *);/* include/linux/mount.h +2
-0@@ -87,6 +87,8 @@ extern bool mnt_may_suid(struct vfsmount *mnt);struct path ;extern struct vfsmount *clone_private_mount (const struct path *path); +extern int __mnt_want_write (struct vfsmount *); +extern void __mnt_drop_write (struct vfsmount *);struct file_system_type ;extern struct vfsmount *fc_mount (struct fs_context *fc);kernel/acct.c +2
-2@@ -227,7 +227,7 @@ static int acct_on(struct filename *pathname)filp_close (file,NULL );return PTR_ERR (internal); } - err =mnt_want_write (internal); + err =__mnt_want_write (internal);if (err) {mntput (internal);kfree (acct);@@ -252,7 +252,7 @@ static int acct_on(struct filename *pathname) old =xchg (&ns->bacct, &acct->pin);mutex_unlock (&acct->lock);pin_kill (old); -mnt_drop_write (mnt); +__mnt_drop_write (mnt);mntput (mnt);return 0 ; }
Quoted 47 lines · Amir Goldstein
Quoted 1 line · Al Viro
...
Quoted 11 lines
OK, so... My first reaction had been complete BS. However, the same goes for your analysis - it's not an ordering problem at all. What happens is that we are replacing
file->path.mntwith a clone and we want the write count contribution (file is opened for write) to be transferred. That's it. We do *NOT* want any kind of freeze protection for the duration of switchover.IOW, the solution is to switch to __mnt_{want,drop}_write() for that switchover; we don't want to mess with freeze protection at all.
Signed-off-by Al Viro <[email protected]>Cool. That works for me.
Thanks for setting this straight.You may add:
Tested-by Amir Goldstein <[email protected]>Reported-by [email protected]Thanks,
Amir.Quoted 21 lines
---
fs/internal.h +0
-2@@ -89,9 +89,7 @@ extern int sb_prepare_remount_readonly(struct super_block *);extern void __init mnt_init (void ); -extern int __mnt_want_write (struct vfsmount *);extern int __mnt_want_write_file (struct file *); -extern void __mnt_drop_write (struct vfsmount *);extern void __mnt_drop_write_file (struct file *);/* include/linux/mount.h +0
-0@@ -87,6 +87,8 @@ extern bool mnt_may_suid(struct vfsmount *mnt);
Quoted 5 lines
include/linux/mount.h extern bool mnt_may_suid(struct vfsmount *mnt); · line 92
Quoted 5 lines
include/linux/mount.h +2
-0struct path ;extern struct vfsmount *clone_private_mount (const struct path *path); +extern int __mnt_want_write (struct vfsmount *); +extern void __mnt_drop_write (struct vfsmount *);
Al,
One minor nit.
If you place these function declarations by their definition order above their wrappers, it would be nicer + patch should apply cleanly to stable v4.4+. As it is, it applies with a minor conflict.
Just a suggestion.
Thanks,
Amir.
Al Viro
maintainerQuoted 3 lines · Amir Goldstein
If new file is on the same fs as old file,
acct_pin_kill(old)fail tofile_start_write_trylock()and skip writing the old file, becausesb_writers(of new) is already taken byacct_on().
The above is BS, BTW. sb_start_write() does *not*
make file_start_write_trylock() to fail. It's basically percpu_down_read() vs. percpu_down_read_trylock().
sb_wait_write() would have file_start_write_trylock() fail (as it should - its caller is freeze_super(); we want an exclusion with attempts to start extra writes there). sb_start_write() sure as hell doesn't - if it would have, we would get its failures from things like e.g. truncate(2) somewhere on the same fs.
We don't want to mess with anything freeze-related in acct_on(), but the bug you are refering to in this part really doesn't exist.