fs/dcache.c - BUG: soft lockup - CPU#5 stuck for 22s! [systemd-udevd:1667]
Al Viro
maintainerQuoted 4 lines · Linus Torvalds
Yeah, I don't think you can reproduce that, but I guess renaming directories into each other (two renames needed) could trigger an ABBA deadlock by changing the topological order of dentry/parent.
Quoted 8 lines
I suspect there's no way in hell that tiny race will ever happen in practice, but let's not risk it.
And your solution (to re-check after just taking the parent lock) seems sufficient and sane, since
dentry_lock_for_move()will always take the parent lock(s) before we move a dentry.So that looks good to me.
BTW, how serious is the problem with __lockref_is_dead(&dentry->d_lockref) with only ->d_parent->d_lock held? From my reading of lib/lockref.c it should be safe - we only do lockref_mark_dead() with ->d_parent->d_lock held, and it'll provide all the serialization and barriers we need.
If I'm right, we could get rid of DCACHE_DENTRY_KILLED completely and replace checking for it with checking for negative ->d_lockref.count. There are two places where we check for it; in shrink_dentry_list() we definitely can go that way (we are holding ->d_lock there) and it simplifies the code nicely. In d_walk(), though (in the bit that used to be try_to_ascend() we only hold ->d_parent->d_lock. It looks like that ought to be safe to replace
if (this_parent != child->d_parent ||
(child->d_flags & DCACHE_DENTRY_KILLED) ||
need_seqretry(&rename_lock, seq)) {
with
if (this_parent != child->d_parent ||
__lockref_is_dead(&child->d_lockref) ||
need_seqretry(&rename_lock, seq)) {and remove DCACHE_DENTRY_KILLED completely...
The other user (in shrink_dentry_list()) simplifies to
if (dentry->d_lockref.count != 0) {
bool can_free = dentry->d_flags & DCACHE_MAY_FREE;
spin_unlock(&dentry->d_lock);
if (parent)
spin_unlock(&parent->d_lock);
if (can_free)
dentry_free(dentry);
continue;
}taking care of both the DCACHE_DENTRY_KILLED case and simple "lazy dget" one, and that one's definitely safe and worth doing.
Would be nice if we could switch d_walk() one as well and kill that flag off, though...
Basically, we havespin_lock(&A);spin_lock(&R.lock);
V = 1;lockref_mark_dead(&R);
...
as the only place where R goes dead and we want to replace spin_lock(&A);
if (V)
... with spin_lock(&A); if (__lockref_is_dead(&R)) ...
Unless I'm missing something subtle in lockref.c, that should be safe... Comments?
Quoted 7 lines · Al Viro
BTW, how serious is the problem with
__lockref_is_dead(&dentry->d_lockref)with only ->d_parent->d_lock held? From my reading oflib/lockref.cit should be safe - we only dolockref_mark_dead()with ->d_parent->d_lock held, and it'll provide all the serialization and barriers we need.If I'm right, we could get rid of
DCACHE_DENTRY_KILLEDcompletely
Yeah, I think that would be good. Except I think you should create a "dentry_is_dead()" helper function that then has that "if you hold the dentry or parent lock, this is safe" comment, because for lockref in general you do need to have the lock in the lockref itself. The fact that dentries have more locking is very much dentry-specific.
But with that, go wild. I'd love to get rid of some of the redundant stuff.
For another example, the
BUG_ON((int)dentry->d_lockref.count > 0);
test makes very little sense any more with lockrefs and the whole dead marker (that should make sure that it never gets incremented), but exists due to the direct conversion.
Linus
Al Viro
maintainerQuoted 5 lines · Linus Torvalds
Yeah, I think that would be good. Except I think you should create a "
dentry_is_dead()" helper function that then has that "if you hold the dentry or parent lock, this is safe" comment, because for lockref in general you do need to have the lock in the lockref itself. The fact that dentries have more locking is very much dentry-specific.
With how many callers? There literally are only two places where we look at that bit; one of them very tempting to convert to ->d_lockref.count != 0, since reactions to positive and negative are very similar. We also have that bogus BUG_ON() you've mentioned (that one simply should die) and only one place where we check it for being negative - autofs4_lookup_active(). And that one is better dealt with by taking removal from their active list from ->d_release() to ->d_prune() (if not turning their ->d_release() into ->d_prune() wholesale) and making ->d_prune() called for hashed and unhashed alike (the only instance *already* checks for d_unhashed() and does nothing in that case; no need to check that in fs/dcache.c). With that done, the check will be gone - all it does is filtering out the ones that are already on the way out, but still hadn't reached ->d_release()).
IOW, it's not a widely used functionality and it's really not something that should be ever needed outside of fs/dcache.c. And in fs/dcache.c we have one call site, so I'm not sure if even mentioning __lockref_not_dead() would make much sense - (int)child->d_lockref.count < 0 might be better, along with a comment about ->d_parent->d_lock serializing it against lockref_mark_dead() in __dentry_kill() just as well as ->d_lock would...
Note that the only reason why autofs is playing those games is that they keep references to dentries that do not contribute to refcount, rip them out when dentry is killed and do that in the wrong method, which opens the window when ->d_lock is already dropped and ->d_release() is inevitable but yet to be called. Solution: rip those references out before dropping ->d_lock, which is what ->d_prune() gives us. To be fair, that code predates ->d_prune() by several years (Jul 2008 and Oct 2011, resp.)
And "vfs: call d_op->d_prune() before unhashing dentry" has added very odd checks for !d_unhashed(), despite ceph ->d_prune() being an explicit no-op in other cases...
While we are at it, what the devil is d_prune_aliases() doing? OK, we grab ->d_lock and see that refcount is 0. Then we call ->d_prune(), bump refcount to 1, forcibly unhash the sucker, drop ->d_lock and ->i_lock and call dput(). Which seems to be far too convoluted... AFAICS, what we want to do is
spin_lock(&dentry->d_lock);
if (!dentry->d_lockref.count) {
parent = lock_parent(dentry);
if (likely(!dentry->d_lockref.count)) {
__dentry_kill(dentry);
goto restart;
}
if (parent)
spin_unlock(&parent->d_lock);
}
spin_unlock(&dentry->d_lock);(which means that pulling ->i_lock trylock into __dentry_kill() is probably not a good plan, more's the pity...) And there goes this second call site of ->d_prune() - after that it would be called exactly in one place, right after __dentry_kill() has done lockref_mark_dead(). The whole reason for calling it there was that forcible unhash used to force dput() to kill the sucker has a side effect of messing ceph ->d_prune()...
Mika Westerberg
Tested-byQuoted 55 lines · Al Viro
Quoted 7 lines · Al Viro
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Al Viro
BTW,
lock_parent()might be better off if in contended case it would not bother withrename_lockand did something like this:
again:Quoted 5 lines
Quoted 1 line
Ack. I think that's much better.
Pushed to #for-linus (with dumb braino fixed - it's if (parent != dentry), not if (parent)). I'll wait with folding it back into the commit that introduces
lock_parent()until we get testing results...Grrr... Sadly, that's not good enough. Leaking
rcu_read_lock()on success is trivial, but there's more serious problem: suppose dentries involved get moved before we get to locking what we thought was parent. We end up taking ->d_lock on two dentries that might be nowhere near each other in the tree, with obvious nasty implications. Would be _very_ hard to reproduce ;-/AFAICS, the following would be safe, but I'd really appreciate any extra eyes on that sucker:
static inline struct dentry *lock_parent(struct dentry *dentry)
{ struct dentry *parent = dentry->d_parent; if (IS_ROOT(dentry)) return NULL; if (likely(spin_trylock(&parent->d_lock))) return parent; spin_unlock(&dentry->d_lock); rcu_read_lock(); again: parent = ACCESS_ONCE(dentry->d_parent); spin_lock(&parent->d_lock); /* * We can't blindly lock dentry until we are sure * that we won't violate the locking order. * While parent->d_lock is not enough to stabilize * dentry->d_parent, it *is* enough to stabilize * dentry->d_parent == parent. */ if (unlikely(parent != dentry->d_parent)) { spin_unlock(&parent->d_lock); goto again; } rcu_read_unlock(); if (parent != dentry) spin_lock(&dentry->d_lock); else parent = NULL; return parent; }
Quoted 3 lines
That variant got force-pushed in place of the previous one, again at the head of #for-linus. And I'm definitely not folding it in until it gets more review and testing.
Tested your latest #for-linus from here:
https://git.kernel.org/cgit/linux/kernel/git/viro/vfs.git/log/?h=for-linus
and the livelock is gone,
Thanks again!
Al Viro
maintainerQuoted 9 lines · Mika Westerberg
Tested your latest #for-linus from here:
https://git.kernel.org/cgit/linux/kernel/git/viro/vfs.git/log/?h=for-linus
and the livelock is gone,
Tested-by Mika Westerberg <[email protected]>Thanks again!
OK... I've just pushed another branch (#for-linus-2) with lock_parent() change folded into the commit that used to introduce rename_lock-based variant. Heads of the branches are byte-for-byte identical:
al@duke:~/linux/trees/vfs$ git log vfs/for-linus|head -1 commit ebc6cb92bfeddf25462842f64604bc9fd2aab5b0 al@duke:~/linux/trees/vfs$ git log vfs/for-linus-2|head -1 commit 8cbf74da435d1bd13dbb790f94c7ff67b2fb6af4 al@duke:~/linux/trees/vfs$ git diff ebc6cb92bfeddf25462842f64604bc9fd2aab5b0 8cbf74da435d1bd13dbb790f94c7ff67b2fb6af4 al@duke:~/linux/trees/vfs$
so testing done on one of them obviously applies to another. FWIW, on #for-linus-2 we have
Shortlog:
Al Viro (6):
lift the "already marked killed" case into shrink_dentry_list() split dentry_kill()
expand dentry_kill(dentry, 0) in shrink_dentry_list() shrink_dentry_list(): take parent's ->d_lock earlier dealing with the rest of shrink_dentry_list() livelock dentry_kill() doesn't need the second argument now
Diffstat:
fs/dcache.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 107 insertions(+), 46 deletions(-)
Linus, how would you prefer it to be handled?
Quoted 2 lines · Al Viro
Linus, how would you prefer it to be handled?
I'll just have to do an rc8. I really hoped to avoid it, because we're going on our family vacation when school is out in two weeks, and it causes problems for the merge window, but it's not like there is much choice - I can't do a 3.15 release with a known regression like that.
So just send me the pull request, and I'll pull it. I'll probably do the "let's increase the x86-64 stack size to 16kB" too, to close _that_ issue as well.
Linus
Al Viro
maintainerQuoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Linus, how would you prefer it to be handled?
I'll just have to do an rc8. I really hoped to avoid it, because we're going on our family vacation when school is out in two weeks, and it causes problems for the merge window, but it's not like there is much choice - I can't do a 3.15 release with a known regression like that.
Sorry about that... ;-/
Quoted 3 lines
So just send me the pull request, and I'll pull it. I'll probably do the "let's increase the x86-64 stack size to 16kB" too, to close _that_ issue as well.
OK, here it is:
Fixes for livelocks in shrink_dentry_list() introduced by fixes to shrink list corruption; the root cause was that trylock of parent's ->d_lock could be disrupted by d_walk() happening on other CPUs, resulting in shrink_dentry_list() making no progress *and* the same d_walk() being called again and again for as long as shrink_dentry_list() doesn't get past that mess. Solution is to have shrink_dentry_list() treat that trylock failure not as "try to do the same thing again", but "lock them in the right order". Please, pull fromgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git for-linus-2
Shortlog:
Al Viro (8):
lift the "already marked killed" case into shrink_dentry_list() split dentry_kill()
expand dentry_kill(dentry, 0) in shrink_dentry_list() shrink_dentry_list(): take parent's ->d_lock earlier dealing with the rest of shrink_dentry_list() livelock dentry_kill() doesn't need the second argument now d_prune_alias(): just lock the parent and call __dentry_kill() dcache.c: call ->d_prune() regardless of d_unhashed()
Diffstat:
fs/dcache.c | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------- 1 file changed, 115 insertions(+), 61 deletions(-)
Al Viro
maintainerQuoted 18 lines · Al Viro
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Linus, how would you prefer it to be handled?
I'll just have to do an rc8. I really hoped to avoid it, because we're going on our family vacation when school is out in two weeks, and it causes problems for the merge window, but it's not like there is much choice - I can't do a 3.15 release with a known regression like that.
Sorry about that... ;-/
Quoted 3 lines
So just send me the pull request, and I'll pull it. I'll probably do the "let's increase the x86-64 stack size to 16kB" too, to close _that_ issue as well.
OK, here it is:
Quoted 9 lines
Fixes for livelocks in
shrink_dentry_list()introduced by fixes to shrink list corruption; the root cause was that trylock of parent's ->d_lock could be disrupted byd_walk()happening on other CPUs, resulting inshrink_dentry_list()making no progress *and* the samed_walk()being called again and again for as long asshrink_dentry_list()doesn't get past that mess. Solution is to haveshrink_dentry_list()treat that trylock failure not as "try to do the same thing again", but "lock them in the right order". Please, pull fromgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.gitfor-linus-2
*GYAH*
Shortlog and diffstat are from the local branch that has two more cleanups on top what's been pushed to vfs.git (and what had been tested). Ones matching what's really in that branch are here:
Shortlog:
Al Viro (6):
lift the "already marked killed" case into shrink_dentry_list() split dentry_kill()
expand dentry_kill(dentry, 0) in shrink_dentry_list() shrink_dentry_list(): take parent's ->d_lock earlier dealing with the rest of shrink_dentry_list() livelock dentry_kill() doesn't need the second argument now
Diffstat:
fs/dcache.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 107 insertions(+), 46 deletions(-)
My apologies - the script I'm using to generate shortlogs takes branch name as an argument, defaulting to HEAD, which was two commits past vfs/for-linus-2. And no, I'm _not_ planning to push that followup stuff until the merge window. Just to make sure: the branch to pull should have head at 8cbf74da435d1bd13dbb790f94c7ff67b2fb6af4 and have the same tree as vfs.git#for-linus, which is what got testing.
Quoted 47 lines · Al Viro
Quoted 18 lines · Al Viro
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Linus, how would you prefer it to be handled?
I'll just have to do an rc8. I really hoped to avoid it, because we're going on our family vacation when school is out in two weeks, and it causes problems for the merge window, but it's not like there is much choice - I can't do a 3.15 release with a known regression like that.
Sorry about that... ;-/
Quoted 3 lines
So just send me the pull request, and I'll pull it. I'll probably do the "let's increase the x86-64 stack size to 16kB" too, to close _that_ issue as well.
OK, here it is:
Quoted 9 lines
Fixes for livelocks in
shrink_dentry_list()introduced by fixes to shrink list corruption; the root cause was that trylock of parent's ->d_lock could be disrupted byd_walk()happening on other CPUs, resulting inshrink_dentry_list()making no progress *and* the samed_walk()being called again and again for as long asshrink_dentry_list()doesn't get past that mess. Solution is to haveshrink_dentry_list()treat that trylock failure not as "try to do the same thing again", but "lock them in the right order". Please, pull fromgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.gitfor-linus-2*GYAH*
Shortlog and diffstat are from the local branch that has two more cleanups on top what's been pushed to vfs.git (and what had been tested). Ones matching what's really in that branch are here:
Shortlog:
Al Viro (6):
lift the "already marked killed" case intoshrink_dentry_list()splitdentry_kill()
expanddentry_kill(dentry, 0)inshrink_dentry_list()shrink_dentry_list(): take parent's ->d_lock earlier dealing with the rest ofshrink_dentry_list()livelockdentry_kill()doesn't need the second argument nowDiffstat:
fs/dcache.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------ 1 file changed, 107 insertions(+), 46 deletions(-)
Quoted 6 lines
My apologies - the script I'm using to generate shortlogs takes branch name as an argument, defaulting to HEAD, which was two commits past vfs/for-linus-2. And no, I'm _not_ planning to push that followup stuff until the merge window. Just to make sure: the branch to pull should have head at
8cbf74da435d1bd13dbb790f94c7ff67b2fb6af4and have the same tree as vfs.git#for-linus, which is what got testing.
One of my machines got the lockdep report below when booting a kernel that contained these patches. This corresponds to Linux v3.15-rc7-102-g1487385edb55.
josh
[ 11.205628] usbcore: registered new interface driver btusb [ 11.212994] systemd-journald[430]: Received request to flush runtime journal from PID 1 [ 11.230780] audit: type=1305 audit(1401544447.755:4): audit_pid=661 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 [ 11.233853] usb 2-1.8.1.1: USB disconnect, device number 6
[ 11.253251] =============================================
[ 11.253254] [ INFO: possible recursive locking detected ]
[ 11.253257] 3.15.0-0.rc7.git4.1.fc21.x86_64 #1 Not tainted
[ 11.253259] ---------------------------------------------
[ 11.253261] systemd-udevd/448 is trying to acquire lock:
[ 11.253264] (&(&dentry->d_lockref.lock)->rlock){+.+...}, at:
[<ffffffff817d66c1>] lock_parent.part.21+0x59/0x69
[ 11.253291]
but task is already holding lock:
[ 11.253295] (&(&dentry->d_lockref.lock)->rlock){+.+...}, at:
[<ffffffff817d669e>] lock_parent.part.21+0x36/0x69
[ 11.253308]
other info that might help us debug this:
[ 11.253312] Possible unsafe locking scenario:[ 11.253317] CPU0 [ 11.253319] ---- [ 11.253322] lock(&(&dentry->d_lockref.lock)->rlock); [ 11.253327] lock(&(&dentry->d_lockref.lock)->rlock); [ 11.253332] *** DEADLOCK ***
[ 11.253337] May be due to missing lock nesting notation
[ 11.253342] 1 lock held by systemd-udevd/448:
[ 11.253345] #0: (&(&dentry->d_lockref.lock)->rlock){+.+...}, at:
[<ffffffff817d669e>] lock_parent.part.21+0x36/0x69
[ 11.253359]
stack backtrace:
[ 11.253366] CPU: 1 PID: 448 Comm: systemd-udevd Not tainted
3.15.0-0.rc7.git4.1.fc21.x86_64 #1
[ 11.253371] Hardware name: Apple Inc.
MacBookPro10,2/Mac-AFD8A9D944EA4843, BIOS
MBP102.88Z.0106.B03.1211161133 11/16/2012
[ 11.253375] 0000000000000000 00000000652a059c ffff88025f71b978
ffffffff817d7dd3
[ 11.253384] ffffffff825a8c60 ffff88025f71ba50 ffffffff810f9084
ffff88003f83d8d8
[ 11.253393] ffffffff825a8c60 0000000000000000 0000000000000000
ffff880200000000
[ 11.253402] Call Trace:
[ 11.253411] [<ffffffff817d7dd3>] dump_stack+0x4d/0x66
[ 11.253422] [<ffffffff810f9084>] __lock_acquire+0x16e4/0x1ca0
[ 11.253434] [<ffffffff81023595>] ? native_sched_clock+0x35/0xa0
[ 11.253443] [<ffffffff810f9e32>] lock_acquire+0xa2/0x1d0
[ 11.253452] [<ffffffff817d66c1>] ? lock_parent.part.21+0x59/0x69
[ 11.253462] [<ffffffff817e0aae>] _raw_spin_lock+0x3e/0x80
[ 11.253470] [<ffffffff817d66c1>] ? lock_parent.part.21+0x59/0x69
[ 11.253478] [<ffffffff817d66c1>] lock_parent.part.21+0x59/0x69
[ 11.253487] [<ffffffff8124b518>] shrink_dentry_list+0x258/0x2a0
[ 11.253495] [<ffffffff8124d2ef>] check_submounts_and_drop+0x8f/0xd0
[ 11.253505] [<ffffffff812bc278>] kernfs_dop_revalidate+0x68/0xe0
[ 11.253514] [<ffffffff8123d881>] lookup_fast+0x331/0x380
[ 11.253523] [<ffffffff8123ef03>] link_path_walk+0x1b3/0x8c0
[ 11.253531] [<ffffffff8123f655>] ? path_lookupat+0x45/0x7b0
[ 11.253539] [<ffffffff8123f67b>] path_lookupat+0x6b/0x7b0
[ 11.253549] [<ffffffff8120d69a>] ? kmem_cache_alloc+0x10a/0x320
[ 11.253557] [<ffffffff8123e22f>] ? getname_flags+0x4f/0x1a0
[ 11.253565] [<ffffffff8123fdeb>] filename_lookup+0x2b/0xc0
[ 11.253574] [<ffffffff81244307>] user_path_at_empty+0x67/0xc0
[ 11.253583] [<ffffffff8123e1b2>] ? final_putname+0x22/0x50
[ 11.253591] [<ffffffff8123e459>] ? putname+0x29/0x40
[ 11.253599] [<ffffffff81244312>] ? user_path_at_empty+0x72/0xc0
[ 11.253607] [<ffffffff81244371>] user_path_at+0x11/0x20
[ 11.253614] [<ffffffff81236fc3>] vfs_fstatat+0x63/0xc0
[ 11.253622] [<ffffffff8123755e>] SYSC_newstat+0x2e/0x60
[ 11.253629] [<ffffffff817eb9d5>] ? sysret_check+0x22/0x5d
[ 11.253637] [<ffffffff810f7525>] ? trace_hardirqs_on_caller+0x105/0x1d0
[ 11.253645] [<ffffffff813d1bfe>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 11.253653] [<ffffffff8123783e>] SyS_newstat+0xe/0x10
[ 11.253662] [<ffffffff817eb9a9>] system_call_fastpath+0x16/0x1b
[ 11.364999] usb 2-1.8.1.2: USB disconnect, device number 7Quoted 3 lines · Josh Boyer
One of my machines got the lockdep report below when booting a kernel that contained these patches.
I think this is just a lacking annotation.
We do nest dentry d_lock locking, and in order to avoid ABBA deadlocks the rule is that we lock things in topological order (parent dentry first). lock_parent() is very careful about that, but doesn't actually tell lockdep about it.
This trivial oneliner should fix it.
Linus
--- fs/dcache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/dcache.c +1 -1
@@ -553,7 +553,7 @@ again: }rcu_read_unlock ();if (parent != dentry) -spin_lock (&dentry->d_lock); +spin_lock_nested (&dentry->d_lock,DENTRY_D_LOCK_NESTED );else parent =NULL ;return parent;
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Josh Boyer
One of my machines got the lockdep report below when booting a kernel that contained these patches.
Quoted 8 lines
I think this is just a lacking annotation.
We do nest dentry
d_locklocking, and in order to avoid ABBA deadlocks the rule is that we lock things in topological order (parent dentry first).lock_parent()is very careful about that, but doesn't actually tell lockdep about it.This trivial oneliner should fix it.
Thanks, I'll give it a spin later today.
josh
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Josh Boyer
One of my machines got the lockdep report below when booting a kernel that contained these patches.
I think this is just a lacking annotation.
Quoted 8 lines
We do nest dentry
d_locklocking, and in order to avoid ABBA deadlocks the rule is that we lock things in topological order (parent dentry first).lock_parent()is very careful about that, but doesn't actually tell lockdep about it.This trivial oneliner should fix it.
Linus
Built the previous kernel with just this applied and it seems to behave correctly on several reboots. I think this looks good. Thanks!
josh
Quoted 18 lines
--- fs/dcache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/dcache.c +1
-1@@ -553,7 +553,7 @@ again: }rcu_read_unlock ();if (parent != dentry) -spin_lock (&dentry->d_lock); +spin_lock_nested (&dentry->d_lock,DENTRY_D_LOCK_NESTED );else parent =NULL ;return parent;
Quoted 39 lines · Al Viro
Quoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Linus, how would you prefer it to be handled?
I'll just have to do an rc8. I really hoped to avoid it, because we're going on our family vacation when school is out in two weeks, and it causes problems for the merge window, but it's not like there is much choice - I can't do a 3.15 release with a known regression like that.
Sorry about that... ;-/
Quoted 3 lines
So just send me the pull request, and I'll pull it. I'll probably do the "let's increase the x86-64 stack size to 16kB" too, to close _that_ issue as well.
OK, here it is:
Fixes for livelocks in
shrink_dentry_list()introduced by fixes to shrink list corruption; the root cause was that trylock of parent's ->d_lock could be disrupted byd_walk()happening on other CPUs, resulting inshrink_dentry_list()making no progress *and* the samed_walk()being called again and again for as long asshrink_dentry_list()doesn't get past that mess. Solution is to haveshrink_dentry_list()treat that trylock failure not as "try to do the same thing again", but "lock them in the right order". Please, pull fromgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.gitfor-linus-2Shortlog:
Al Viro (8):
lift the "already marked killed" case intoshrink_dentry_list()splitdentry_kill()
expanddentry_kill(dentry, 0)inshrink_dentry_list()shrink_dentry_list(): take parent's ->d_lock earlier dealing with the rest ofshrink_dentry_list()livelockdentry_kill()doesn't need the second argument nowd_prune_alias(): just lock the parent and call__dentry_kill()dcache.c: call ->d_prune() regardless ofd_unhashed()
Quoted 4 lines
Diffstat:
fs/dcache.c | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------------------------- 1 file changed, 115 insertions(+), 61 deletions(-)
Did you push all?
$ git log --oneline v3.15-rc7.. 8cbf74d dentry_kill() doesn't need the second argument now b2b8019 dealing with the rest of shrink_dentry_list() livelock 046b961 shrink_dentry_list(): take parent's ->d_lock earlier ff2fde9 expand dentry_kill(dentry, 0) in shrink_dentry_list() e55fd01 split dentry_kill() 64fd72e lift the "already marked killed" case into shrink_dentry_list() b6dd6f4 vfs: fix vmplice_to_user()
- Sedat -
Quoted 2 lines · Al Viro
Here's an updated patch, hopefully slightly less vomit-inducing.
Hmm. Less vomit-inducing, except for this part:
Quoted 8 lines
dentry = list_entry(list->prev, struct dentry, d_lru);fs/dcache.c +6
-0+ + parent =NULL ;spin_lock (&dentry->d_lock); +if (!IS_ROOT (dentry)) { + parent = dentry->d_parent; +if (unlikely (!spin_trylock (&parent->d_lock))) { +spin_unlock (&dentry->d_lock);
fs/dcache.c static void shrink_dentry_list(struct list_head *list) · line 819
Quoted 10 lines
fs/dcache.c +10
-0+ parent =NULL ; +read_seqlock_excl (&rename_lock); +if (!IS_ROOT (dentry)) { + parent = dentry->d_parent; +spin_lock (&parent->d_lock); + } +read_sequnlock_excl (&rename_lock); +spin_lock (&dentry->d_lock); + } + }
Ugh, that just *screams* for a helper function. Something like
parent = get_parent_and_lock(dentry);
or whatever, with that trylock/renamelock dance separated out. The rule would be that it would lock the "dentry", and return the (similarly locked) parent. Or NULL for a root dentry, of course.
Please?
Linus
Al Viro
maintainerQuoted 8 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Here's an updated patch, hopefully slightly less vomit-inducing.
Hmm. Less vomit-inducing, except for this part:
Ugh, that just *screams* for a helper function. Something like
Quoted 7 lines
parent = get_parent_and_lock(dentry);or whatever, with that trylock/renamelock dance separated out. The rule would be that it would lock the "dentry", and return the (similarly locked) parent. Or NULL for a root dentry, of course.
Please?
Already done in my current tree; see below for the next commit in there... I can separate that helper from the rest (dealing with the same livelock for dentry_kill(dentry, 1) caller in shrink_dentry_list()). All of that is very much subject to reordering, resplitting, writing more decent commit messages, etc.
commit 67e2554f6ca62e20363139f6f2968063410f0f3b
Author: Al Viro <[email protected]> Date: Wed May 28 23:53:36 2014 -0400
deal with "put the parent" side of shrink_dentry_list()
fs/dcache.c +40 -21
@@ -793,30 +793,31 @@ restart: }EXPORT_SYMBOL (d_prune_aliases); -static void shrink_dentry_list (struct list_head *list) +static inline struct dentry *lock_parent (struct dentry *dentry) { -struct dentry *dentry, *parent; - -while (!list_empty (list)) { -struct inode *inode; - dentry = list_entry(list->prev, struct dentry, d_lru); - +struct dentry *parent = dentry->d_parent; +if (IS_ROOT (dentry)) +return NULL ; +if (unlikely (!spin_trylock (&parent->d_lock))) { +spin_unlock (&dentry->d_lock); +read_seqlock_excl (&rename_lock); parent =NULL ; -spin_lock (&dentry->d_lock);if (!IS_ROOT (dentry)) { parent = dentry->d_parent; -if (unlikely (!spin_trylock (&parent->d_lock))) { -spin_unlock (&dentry->d_lock); - parent =NULL ; -read_seqlock_excl (&rename_lock); -if (!IS_ROOT (dentry)) { - parent = dentry->d_parent; -spin_lock (&parent->d_lock); - } -read_sequnlock_excl (&rename_lock); -spin_lock (&dentry->d_lock); - } +spin_lock (&parent->d_lock); } +read_sequnlock_excl (&rename_lock); +spin_lock (&dentry->d_lock); + } +return parent; +} + +static void shrink_dentry_list (struct list_head *list) +{ +while (!list_empty (list)) { +struct inode *inode; + struct dentry *dentry = list_entry(list->prev, struct dentry, d_lru); +struct dentry *parent =lock_parent (dentry);/* * The dispose list is isolated and dentries are not accounted
@@ -864,8 +865,26 @@ static void shrink_dentry_list(struct list_head *list)* fragmentation. */ dentry = parent; -while (dentry && !lockref_put_or_lock (&dentry->d_lockref)) - dentry =dentry_kill (dentry); +while (dentry && !lockref_put_or_lock (&dentry->d_lockref)) { + parent =lock_parent (dentry); +if (dentry->d_lockref.count !=1 ) { + dentry->d_lockref.count--; +spin_unlock (&dentry->d_lock); +if (parent) +spin_unlock (&parent->d_lock); +break ; + } + inode = dentry->d_inode;/* can't be NULL */ +if (unlikely (!spin_trylock (&inode->i_lock))) { +spin_unlock (&dentry->d_lock); +if (parent) +spin_unlock (&parent->d_lock); +cpu_relax (); +continue ; + } +__dentry_kill (dentry); + dentry = parent; + } } }
Al Viro
maintainerQuoted 15 lines · Al Viro
Quoted 14 lines · Linus Torvalds
Quoted 2 lines · Al Viro
Here's an updated patch, hopefully slightly less vomit-inducing.
Hmm. Less vomit-inducing, except for this part:
Ugh, that just *screams* for a helper function. Something like
parent = get_parent_and_lock(dentry);or whatever, with that trylock/renamelock dance separated out. The rule would be that it would lock the "dentry", and return the (similarly locked) parent. Or NULL for a root dentry, of course.
Quoted 7 lines
Quoted 1 line
Please?
Already done in my current tree; see below for the next commit in there... I can separate that helper from the rest (dealing with the same livelock for
dentry_kill(dentry, 1)caller inshrink_dentry_list()). All of that is very much subject to reordering, resplitting, writing more decent commit messages, etc.
Except that it's from the wrong branch ;-/ Correct one follows:
commit fb860957449c07c6f1a1dd911e83a635b6f11f21
Author: Al Viro <[email protected]> Date: Thu May 29 00:37:49 2014 -0400
deal with "put the parent" side of shrink_dentry_list()
fs/dcache.c +42 -20
@@ -793,30 +793,34 @@ restart: }EXPORT_SYMBOL (d_prune_aliases); -static void shrink_dentry_list (struct list_head *list) +static inline struct dentry *lock_parent (struct dentry *dentry) { -struct dentry *dentry, *parent; +struct dentry *parent = dentry->d_parent; +if (IS_ROOT (dentry)) +return NULL ; +if (unlikely (!spin_trylock (&parent->d_lock))) { +spin_unlock (&dentry->d_lock); +read_seqlock_excl (&rename_lock); + parent =NULL ; +if (!IS_ROOT (dentry)) { + parent = dentry->d_parent; +spin_lock (&parent->d_lock); + } +read_sequnlock_excl (&rename_lock); +spin_lock (&dentry->d_lock); + } +return parent; +} +static void shrink_dentry_list (struct list_head *list) +{while (!list_empty (list)) {struct inode *inode; - dentry = list_entry(list->prev, struct dentry, d_lru); + struct dentry *dentry = list_entry(list->prev, struct dentry, d_lru); +struct dentry *parent; - parent =NULL ;spin_lock (&dentry->d_lock); -if (!IS_ROOT (dentry)) { - parent = dentry->d_parent; -if (unlikely (!spin_trylock (&parent->d_lock))) { -spin_unlock (&dentry->d_lock); - parent =NULL ; -read_seqlock_excl (&rename_lock); -if (!IS_ROOT (dentry)) { - parent = dentry->d_parent; -spin_lock (&parent->d_lock); - } -read_sequnlock_excl (&rename_lock); -spin_lock (&dentry->d_lock); - } - } + parent =lock_parent (dentry);/* * The dispose list is isolated and dentries are not accounted
@@ -864,8 +868,26 @@ static void shrink_dentry_list(struct list_head *list)* fragmentation. */ dentry = parent; -while (dentry && !lockref_put_or_lock (&dentry->d_lockref)) - dentry =dentry_kill (dentry); +while (dentry && !lockref_put_or_lock (&dentry->d_lockref)) { + parent =lock_parent (dentry); +if (dentry->d_lockref.count !=1 ) { + dentry->d_lockref.count--; +spin_unlock (&dentry->d_lock); +if (parent) +spin_unlock (&parent->d_lock); +break ; + } + inode = dentry->d_inode;/* can't be NULL */ +if (unlikely (!spin_trylock (&inode->i_lock))) { +spin_unlock (&dentry->d_lock); +if (parent) +spin_unlock (&parent->d_lock); +cpu_relax (); +continue ; + } +__dentry_kill (dentry); + dentry = parent; + } } }