Skip to contentSkip to conversation

i[RESEND] kernel/acct.c: fix locking order when switching acct files

PATCH In review Amir Goldstein 1 participant 3 messages Sep 29, 2018 – Feb 26, 2019 VFS

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i[RESEND] kernel/acct.c: fix locking order when switching acct files

This looks like an old bug, pre-dating the "Fixes" commit, but the "Fixes" commit did not handle it properly.

The bug recently surfaced as a lockdep possible deadlock warning with commit d1d04ef8572b ("ovl: stack file ops").

When acct_on() replaces one acct file with another, it takes sb_writers lock on new file sb and calls acct_pin_kill(old) before releasing the sb_writers lock.

If new file is on the same fs as old file, acct_pin_kill(old) fail to file_start_write_trylock() and skip writing the old file, because sb_writers (of new) is already taken by acct_on().

If new file is not on same fs as old file, this ordering violation creates an unneeded dependency between new sb_writers and old sb_writers, which may later be reported as possible deadlock.

This could result in an actual deadlock if acct file is replaced from an old file in overlayfs over "real fs" to a new file in "real fs". acct_on() takes freeze protection on "real fs" and tries to write to overlayfs file. overlayfs is not freeze protected so do_acct_process() can carry on with __kernel_write() to overlayfs file, which would try to take freeze protection on "real fs" and deadlock.

Reproducer of lockdep possible deadlock warning:

  ./run --ov -s # unionmount-testsuite
  touch /mnt/x /upper/y
  accton /mnt/x
  accton /upper/y

======================================================
WARNING: possible circular locking dependency detected 4.19.0-rc1-xfstests #3424 Not tainted
------------------------------------------------------
accton/1390 is trying to acquire lock:
00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76

but task is already holding lock:
000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42

Reported-by [email protected]
Fixes 59eda0e07f43 ("new fs_pin killing logics")

---

Al,

Welcome back. It would be nice to get an ACK (or an Applied) on this patch. Fixes label claims to fix your commit, but I believe the bug was already there before your commit.

As you can see, I have a reproducer to demonstrate the manifestation of the bug in the case of switching acct file from overlayfs to real fs. This started to manifest with overlayfs stacked f_op.
I have made another claim which seems obvious from the code about changing acct file on same fs, but did not bother to write a reproducer for that case.

Thanks,
Amir.

 kernel/acct.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/acct.c +1 -1
index addf7732fb56..c09355a7ae46 100644
@@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
 	rcu_read_lock();
 	old = xchg(&ns->bacct, &acct->pin);
 	mutex_unlock(&acct->lock);
-	pin_kill(old);
 	mnt_drop_write(mnt);
+	pin_kill(old);
 	mntput(mnt);
 	return 0;
 }
Signature · 1 line
2.17.1
#1 lore raw

Re: [PATCH][RESEND] kernel/acct.c: fix locking order when switching acct files

Quoted 61 lines · Amir Goldstein

This looks like an old bug, pre-dating the "Fixes" commit, but the "Fixes" commit did not handle it properly.

The bug recently surfaced as a lockdep possible deadlock warning with commit d1d04ef8572b ("ovl: stack file ops").

When acct_on() replaces one acct file with another, it takes sb_writers lock on new file sb and calls acct_pin_kill(old) before releasing the sb_writers lock.

If new file is on the same fs as old file, acct_pin_kill(old) fail to file_start_write_trylock() and skip writing the old file, because sb_writers (of new) is already taken by acct_on().

If new file is not on same fs as old file, this ordering violation creates an unneeded dependency between new sb_writers and old sb_writers, which may later be reported as possible deadlock.

This could result in an actual deadlock if acct file is replaced from an old file in overlayfs over "real fs" to a new file in "real fs". acct_on() takes freeze protection on "real fs" and tries to write to overlayfs file. overlayfs is not freeze protected so do_acct_process() can carry on with __kernel_write() to overlayfs file, which would try to take freeze protection on "real fs" and deadlock.

Reproducer of lockdep possible deadlock warning:

  ./run --ov -s # unionmount-testsuite
  touch /mnt/x /upper/y
  accton /mnt/x
  accton /upper/y

======================================================
WARNING: possible circular locking dependency detected 4.19.0-rc1-xfstests #3424 Not tainted
------------------------------------------------------
accton/1390 is trying to acquire lock:
00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76

but task is already holding lock:
000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42

Reported-by [email protected]
Fixes 59eda0e07f43 ("new fs_pin killing logics")

---

Al,

Welcome back. It would be nice to get an ACK (or an Applied) on this patch. Fixes label claims to fix your commit, but I believe the bug was already there before your commit.

As you can see, I have a reproducer to demonstrate the manifestation of the bug in the case of switching acct file from overlayfs to real fs. This started to manifest with overlayfs stacked f_op.
I have made another claim which seems obvious from the code about changing acct file on same fs, but did not bother to write a reproducer for that case.

Quoted 2 lines

Thanks,
Amir.

PING

Quoted 21 lines
 kernel/acct.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/acct.c +1 -2
index addf7732fb56..c09355a7ae46 100644
@@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
        rcu_read_lock();
        old = xchg(&ns->bacct, &acct->pin);
        mutex_unlock(&acct->lock);
-       pin_kill(old);
        mnt_drop_write(mnt);
+       pin_kill(old);
        mntput(mnt);
        return 0;
 }
--

2.17.1

#2 lore raw

Re: [PATCH][RESEND] kernel/acct.c: fix locking order when switching acct files

Quoted 63 lines · Amir Goldstein
Quoted 61 lines · Amir Goldstein

This looks like an old bug, pre-dating the "Fixes" commit, but the "Fixes" commit did not handle it properly.

The bug recently surfaced as a lockdep possible deadlock warning with commit d1d04ef8572b ("ovl: stack file ops").

When acct_on() replaces one acct file with another, it takes sb_writers lock on new file sb and calls acct_pin_kill(old) before releasing the sb_writers lock.

If new file is on the same fs as old file, acct_pin_kill(old) fail to file_start_write_trylock() and skip writing the old file, because sb_writers (of new) is already taken by acct_on().

If new file is not on same fs as old file, this ordering violation creates an unneeded dependency between new sb_writers and old sb_writers, which may later be reported as possible deadlock.

This could result in an actual deadlock if acct file is replaced from an old file in overlayfs over "real fs" to a new file in "real fs". acct_on() takes freeze protection on "real fs" and tries to write to overlayfs file. overlayfs is not freeze protected so do_acct_process() can carry on with __kernel_write() to overlayfs file, which would try to take freeze protection on "real fs" and deadlock.

Reproducer of lockdep possible deadlock warning:

  ./run --ov -s # unionmount-testsuite
  touch /mnt/x /upper/y
  accton /mnt/x
  accton /upper/y

======================================================
WARNING: possible circular locking dependency detected 4.19.0-rc1-xfstests #3424 Not tainted
------------------------------------------------------
accton/1390 is trying to acquire lock:
00000000e0585aa5 (&acct->lock#2){+.+.}, at: acct_pin_kill+0x1b/0x76

but task is already holding lock:
000000003692505a (sb_writers#6){.+.+}, at: mnt_want_write+0x1d/0x42

Reported-by [email protected]
Fixes 59eda0e07f43 ("new fs_pin killing logics")

---

Al,

Welcome back. It would be nice to get an ACK (or an Applied) on this patch. Fixes label claims to fix your commit, but I believe the bug was already there before your commit.

As you can see, I have a reproducer to demonstrate the manifestation of the bug in the case of switching acct file from overlayfs to real fs. This started to manifest with overlayfs stacked f_op.
I have made another claim which seems obvious from the code about changing acct file on same fs, but did not bother to write a reproducer for that case.

Quoted 4 lines
Quoted 2 lines

Thanks,
Amir.

PING

PING^2

Quoted 22 lines
Quoted 22 lines

 kernel/acct.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/acct.c +1 -2
index addf7732fb56..c09355a7ae46 100644
@@ -251,8 +251,8 @@ static int acct_on(struct filename *pathname)
        rcu_read_lock();
        old = xchg(&ns->bacct, &acct->pin);
        mutex_unlock(&acct->lock);
-       pin_kill(old);
        mnt_drop_write(mnt);
+       pin_kill(old);
        mntput(mnt);
        return 0;
 }
--

2.17.1