Skip to contentSkip to conversation

fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

PATCH Superseded Will Deacon 3 participants 4 messages Feb 13, 2018 – Feb 15, 2018 VFS

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs: dcache: Avoid livelock between d_alloc_parallel and __d_add

If d_alloc_parallel runs concurrently with __d_add, it is possible for d_alloc_parallel to continuously retry whilst i_dir_seq has been incremented to an odd value by __d_add:

CPU0:
__d_add

	n = start_dir_add(dir);
		cmpxchg(&dir->i_dir_seq, n, n + 1) == n

CPU1:
d_alloc_parallel
retry:

	seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
	hlist_bl_lock(b);
		bit_spin_lock(0, (unsigned long *)b); // Always succeeds

CPU0:

	__d_lookup_done(dentry)
		hlist_bl_lock
			bit_spin_lock(0, (unsigned long *)b); // Never succeeds

CPU1:

	if (unlikely(parent->d_inode->i_dir_seq != seq)) {
		hlist_bl_unlock(b);
		goto retry;
	}

Since the simple bit_spin_lock used to implement hlist_bl_lock does not provide any fairness guarantees, then CPU1 can starve CPU0 of the lock and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot exit its retry loop because the sequence number always has the bottom bit set.

This patch resolves the livelock by not taking hlist_bl_lock in d_alloc_parallel if the sequence counter is odd, since any subsequent masked comparison with i_dir_seq will fail anyway.

Cc Peter Zijlstra <[email protected]>
Cc Al Viro <[email protected]>
---
 fs/dcache.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
fs/dcache.c +7 -1
index 7c38f39958bc..b243deec298c 100644
@@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
 
 retry:
 	rcu_read_lock();
-	seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
+	seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
 	r_seq = read_seqbegin(&rename_lock);
 	dentry = __d_lookup_rcu(parent, name, &d_seq);
 	if (unlikely(dentry)) {
@@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
 		rcu_read_unlock();
 		goto retry;
 	}
+
+	if (unlikely(seq & 1)) {
+		rcu_read_unlock();
+		goto retry;
+	}
+
 	hlist_bl_lock(b);
 	if (unlikely(parent->d_inode->i_dir_seq != seq)) {
 		hlist_bl_unlock(b);
Signature · 1 line
2.1.4
Acked-by on v1 1/1
#1 lore raw
Quoted 21 lines · Will Deacon

If d_alloc_parallel runs concurrently with __d_add, it is possible for d_alloc_parallel to continuously retry whilst i_dir_seq has been incremented to an odd value by __d_add:

CPU0:
__d_add

	n = start_dir_add(dir);
		cmpxchg(&dir->i_dir_seq, n, n + 1) == n

CPU1:
d_alloc_parallel
retry:

	seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
	hlist_bl_lock(b);
		bit_spin_lock(0, (unsigned long *)b); // Always succeeds

CPU0:

	__d_lookup_done(dentry)
		hlist_bl_lock
			bit_spin_lock(0, (unsigned long *)b); // Never succeeds
Quoted 7 lines

CPU1:

	if (unlikely(parent->d_inode->i_dir_seq != seq)) {
		hlist_bl_unlock(b);
		goto retry;
	}

Since the simple bit_spin_lock used to implement hlist_bl_lock does not

And cannot, a single bit is just not enough state.

Quoted 9 lines

provide any fairness guarantees, then CPU1 can starve CPU0 of the lock and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot exit its retry loop because the sequence number always has the bottom bit set.

This patch resolves the livelock by not taking hlist_bl_lock in d_alloc_parallel if the sequence counter is odd, since any subsequent masked comparison with i_dir_seq will fail anyway.

Quoted 21 lines
---
 fs/dcache.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
fs/dcache.c +1 -1
index 7c38f39958bc..b243deec298c 100644
@@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
 
 retry:
 	rcu_read_lock();
-	seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
+	seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
 	r_seq = read_seqbegin(&rename_lock);
 	dentry = __d_lookup_rcu(parent, name, &d_seq);
 	if (unlikely(dentry)) {
@@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
 		rcu_read_unlock();

fs/dcache.c struct dentry *d_alloc_parallel(struct dentry *parent, · line 2505

Quoted 10 lines
fs/dcache.c +6 -0
 		goto retry;
 	}
+
+	if (unlikely(seq & 1)) {
+		rcu_read_unlock();
+		goto retry;
+	}
+
 	hlist_bl_lock(b);
 	if (unlikely(parent->d_inode->i_dir_seq != seq)) {

Also, should that not read:

	if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {

I mean, load-tearing can only result in additional failure, but still.

Quoted 1 line
fs/dcache.c +0 -0
 		hlist_bl_unlock(b);
Reviewed-by on v1 1/1
#2 lore raw
In reply to Will Deacon
Quoted 7 lines · Will Deacon

This patch resolves the livelock by not taking hlist_bl_lock in d_alloc_parallel if the sequence counter is odd, since any subsequent masked comparison with i_dir_seq will fail anyway.

Cc Peter Zijlstra <[email protected]>
Cc Al Viro <[email protected]>

I wonder whether it makes sense to turn i_dir_seq into a seqcount_t, which would give us the lockdep checking as well.

#3 lore raw

Hi Matthew,

Quoted 10 lines · Matthew Wilcox
Quoted 7 lines · Will Deacon

This patch resolves the livelock by not taking hlist_bl_lock in d_alloc_parallel if the sequence counter is odd, since any subsequent masked comparison with i_dir_seq will fail anyway.

Cc Peter Zijlstra <[email protected]>
Cc Al Viro <[email protected]>

Thanks!

Quoted 2 lines

I wonder whether it makes sense to turn i_dir_seq into a seqcount_t, which would give us the lockdep checking as well.

I'm not sure it's quite as simple as that. start_dir_add looks very much like it's intended to run concurrently, so we'd need a write_seqcount implementation that provides the same atomicity guarantees.

Will