fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
PATCH 1/1 fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
If d_alloc_parallel runs concurrently with __d_add, it is possible for d_alloc_parallel to continuously retry whilst i_dir_seq has been incremented to an odd value by __d_add:
CPU0:__d_add
n = start_dir_add(dir); cmpxchg(&dir->i_dir_seq, n, n + 1) == n
CPU1:d_alloc_parallel
retry:
seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1; hlist_bl_lock(b); bit_spin_lock(0, (unsigned long *)b); // Always succeeds
CPU0:
__d_lookup_done(dentry) hlist_bl_lock bit_spin_lock(0, (unsigned long *)b); // Never succeeds
CPU1:
if (unlikely(parent->d_inode->i_dir_seq != seq)) {
hlist_bl_unlock(b);
goto retry;
}Since the simple bit_spin_lock used to implement hlist_bl_lock does not provide any fairness guarantees, then CPU1 can starve CPU0 of the lock and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot exit its retry loop because the sequence number always has the bottom bit set.
This patch resolves the livelock by not taking hlist_bl_lock in d_alloc_parallel if the sequence counter is odd, since any subsequent masked comparison with i_dir_seq will fail anyway.
--- fs/dcache.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
fs/dcache.c +7 -1
@@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent, retry:rcu_read_lock (); - seq =smp_load_acquire (&parent->d_inode->i_dir_seq) & ~1 ; + seq =smp_load_acquire (&parent->d_inode->i_dir_seq); r_seq =read_seqbegin (&rename_lock); dentry =__d_lookup_rcu (parent, name, &d_seq);if (unlikely (dentry)) {
@@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,rcu_read_unlock (); goto retry; } + +if (unlikely (seq &1 )) { +rcu_read_unlock (); + goto retry; + } +hlist_bl_lock (b);if (unlikely (parent->d_inode->i_dir_seq != seq)) {hlist_bl_unlock (b);
Signature · 1 line
2.1.4
Quoted 21 lines · Will Deacon
If
d_alloc_parallelruns concurrently with__d_add, it is possible ford_alloc_parallelto continuously retry whilsti_dir_seqhas been incremented to an odd value by__d_add:CPU0:
__d_addn = start_dir_add(dir); cmpxchg(&dir->i_dir_seq, n, n + 1) == nCPU1:
d_alloc_parallel
retry:seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1; hlist_bl_lock(b); bit_spin_lock(0, (unsigned long *)b); // Always succeedsCPU0:
__d_lookup_done(dentry) hlist_bl_lock bit_spin_lock(0, (unsigned long *)b); // Never succeeds
Quoted 7 lines
CPU1:
if (unlikely(parent->d_inode->i_dir_seq != seq)) { hlist_bl_unlock(b); goto retry; }Since the simple
bit_spin_lockused to implementhlist_bl_lockdoes not
And cannot, a single bit is just not enough state.
Quoted 9 lines
provide any fairness guarantees, then CPU1 can starve CPU0 of the lock and prevent it from reaching
end_dir_add(dir), therefore CPU1 cannot exit its retry loop because the sequence number always has the bottom bit set.This patch resolves the livelock by not taking
hlist_bl_lockind_alloc_parallelif the sequence counter is odd, since any subsequent masked comparison withi_dir_seqwill fail anyway.
Quoted 21 lines
Cc Al Viro <[email protected]>Signed-off-by Will Deacon <[email protected]>--- fs/dcache.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)fs/dcache.c +1
-1@@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent, retry:rcu_read_lock (); - seq =smp_load_acquire (&parent->d_inode->i_dir_seq) & ~1 ; + seq =smp_load_acquire (&parent->d_inode->i_dir_seq); r_seq =read_seqbegin (&rename_lock); dentry =__d_lookup_rcu (parent, name, &d_seq);if (unlikely (dentry)) {@@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,rcu_read_unlock ();
fs/dcache.c struct dentry *d_alloc_parallel(struct dentry *parent, · line 2505
Quoted 10 lines
fs/dcache.c +6
-0goto retry; } + +if (unlikely (seq &1 )) { +rcu_read_unlock (); + goto retry; + } +hlist_bl_lock (b);if (unlikely (parent->d_inode->i_dir_seq != seq)) {
Also, should that not read:
if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {I mean, load-tearing can only result in additional failure, but still.
Quoted 1 line
fs/dcache.c +0
-0hlist_bl_unlock (b);
Quoted 7 lines · Will Deacon
This patch resolves the livelock by not taking
hlist_bl_lockind_alloc_parallelif the sequence counter is odd, since any subsequent masked comparison withi_dir_seqwill fail anyway.Cc Peter Zijlstra <[email protected]>Cc Al Viro <[email protected]>Signed-off-by Will Deacon <[email protected]>
I wonder whether it makes sense to turn i_dir_seq into a seqcount_t, which would give us the lockdep checking as well.
Hi Matthew,
Quoted 10 lines · Matthew Wilcox
Quoted 7 lines · Will Deacon
This patch resolves the livelock by not taking
hlist_bl_lockind_alloc_parallelif the sequence counter is odd, since any subsequent masked comparison withi_dir_seqwill fail anyway.Cc Peter Zijlstra <[email protected]>Cc Al Viro <[email protected]>Signed-off-by Will Deacon <[email protected]>Reviewed-by Matthew Wilcox <[email protected]>
Thanks!
Quoted 2 lines
I wonder whether it makes sense to turn
i_dir_seqinto aseqcount_t, which would give us the lockdep checking as well.
I'm not sure it's quite as simple as that. start_dir_add looks very much like it's intended to run concurrently, so we'd need a write_seqcount implementation that provides the same atomicity guarantees.
Will