Skip to contentSkip to conversation

bcachefs: Fix lost wake up

PATCH In review Alan Huang 2 participants 5 messages Aug 27, 2024 – Sep 1, 2024

lore
Share & tools
lore

Alan Huang

author
#0 lore raw

PATCH 1/1 bcachefs: Fix lost wake up

If the reader acquires the read lock and then the writer enters the slow path, while the reader proceeds to the unlock path, the following scenario can occur without the change:

writer: pcpu_read_count(lock) return 1 (so __do_six_trylock will return 0)
reader: this_cpu_dec(*lock->readers)
reader: smp_mb()
reader: state = atomic_read(&lock->state) (there is no waiting flag set)
writer: six_set_bitmask()

then the writer will sleep forever.

---
 fs/bcachefs/six.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
fs/bcachefs/six.c +9 -3
index 3a494c5d1247..b3583c50ef04 100644
@@ -169,11 +169,17 @@ static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type,
 				ret = -1 - SIX_LOCK_write;
 		}
 	} else if (type == SIX_LOCK_write && lock->readers) {
-		if (try) {
+		if (try)
 			atomic_add(SIX_LOCK_HELD_write, &lock->state);
-			smp_mb__after_atomic();
-		}
 
+		/*
+		 * Make sure atomic_add happens before pcpu_read_count and
+		 * six_set_bitmask in slow path happens before pcpu_read_count.
+		 *
+		 * Paired with the smp_mb() in read lock fast path (per-cpu mode)
+		 * and the one before atomic_read in read unlock path.
+		 */
+		smp_mb();
 		ret = !pcpu_read_count(lock);
 
 		if (try && !ret) {
Signature · 1 line
2.45.2
#1 lore raw
Quoted 29 lines · Alan Huang

If the reader acquires the read lock and then the writer enters the slow path, while the reader proceeds to the unlock path, the following scenario can occur without the change:

writer: pcpu_read_count(lock) return 1 (so __do_six_trylock will return 0)
reader: this_cpu_dec(*lock->readers)
reader: smp_mb()
reader: state = atomic_read(&lock->state) (there is no waiting flag set)
writer: six_set_bitmask()

then the writer will sleep forever.

---
 fs/bcachefs/six.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
fs/bcachefs/six.c +1 -2
index 3a494c5d1247..b3583c50ef04 100644
@@ -169,11 +169,17 @@ static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type,
 				ret = -1 - SIX_LOCK_write;
 		}
 	} else if (type == SIX_LOCK_write && lock->readers) {
-		if (try) {
+		if (try)
 			atomic_add(SIX_LOCK_HELD_write, &lock->state);
-			smp_mb__after_atomic();

fs/bcachefs/six.c static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type, · line 182

Quoted 10 lines
fs/bcachefs/six.c +8 -1
-		}
 
+		/*
+		 * Make sure atomic_add happens before pcpu_read_count and
+		 * six_set_bitmask in slow path happens before pcpu_read_count.
+		 *
+		 * Paired with the smp_mb() in read lock fast path (per-cpu mode)
+		 * and the one before atomic_read in read unlock path.
+		 */
+		smp_mb();

What is this barrier ordering?

This is only a change in the !try case, and we didn't do anything before checking pcpu_read_count() - except way back in six_lock_slowpath() where we set SIX_LOCK_HELD_write, but that does have a barrier.

Quoted 6 lines
fs/bcachefs/six.c +0 -0
 		ret = !pcpu_read_count(lock);
 
 		if (try && !ret) {
Signature · 1 line
2.45.2
author
#2 lore raw
Quoted 45 lines

2024年9月1日 09:09,Kent Overstreet <[email protected]> wrote:

Quoted 29 lines · Alan Huang

If the reader acquires the read lock and then the writer enters the slow path, while the reader proceeds to the unlock path, the following scenario can occur without the change:

writer: pcpu_read_count(lock) return 1 (so __do_six_trylock will return 0)
reader: this_cpu_dec(*lock->readers)
reader: smp_mb()
reader: state = atomic_read(&lock->state) (there is no waiting flag set)
writer: six_set_bitmask()

then the writer will sleep forever.

---
fs/bcachefs/six.c | 12 +++++++++---

1 file changed, 9 insertions(+), 3 deletions(-)
fs/bcachefs/six.c +1 -2
index 3a494c5d1247..b3583c50ef04 100644
@@ -169,11 +169,17 @@ static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type,
 ret = -1 - SIX_LOCK_write;
 }
 } else if (type == SIX_LOCK_write && lock->readers) {
- if (try) {
+ if (try)
 atomic_add(SIX_LOCK_HELD_write, &lock->state);
- smp_mb__after_atomic();

fs/bcachefs/six.c static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type, · line 182

Quoted 10 lines
fs/bcachefs/six.c +8 -1
- }
 
+ /*
+  * Make sure atomic_add happens before pcpu_read_count and
+  * six_set_bitmask in slow path happens before pcpu_read_count.
+  *
+  * Paired with the smp_mb() in read lock fast path (per-cpu mode)
+  * and the one before atomic_read in read unlock path.
+  */
+ smp_mb();

What is this barrier ordering?

Quoted 3 lines

This is only a change in the !try case, and we didn't do anything before checking pcpu_read_count() - except way back in six_lock_slowpath() where we set SIX_LOCK_HELD_write, but that does have a barrier.

Make sure six_set_bitmask(lock, SIX_LOCK_WAITING_read << type)

happens before pcpu_read_count()

Quoted 6 lines
Quoted 6 lines
fs/bcachefs/six.c +0 -0
 ret = !pcpu_read_count(lock);
 
 if (try && !ret) {
Signature · 1 line
2.45.2
#3 lore raw
Quoted 45 lines · Alan Huang
Quoted 45 lines

2024年9月1日 09:09,Kent Overstreet <[email protected]> wrote:

Quoted 29 lines · Alan Huang

If the reader acquires the read lock and then the writer enters the slow path, while the reader proceeds to the unlock path, the following scenario can occur without the change:

writer: pcpu_read_count(lock) return 1 (so __do_six_trylock will return 0)
reader: this_cpu_dec(*lock->readers)
reader: smp_mb()
reader: state = atomic_read(&lock->state) (there is no waiting flag set)
writer: six_set_bitmask()

then the writer will sleep forever.

---
fs/bcachefs/six.c | 12 +++++++++---

1 file changed, 9 insertions(+), 3 deletions(-)
fs/bcachefs/six.c +1 -2
index 3a494c5d1247..b3583c50ef04 100644
@@ -169,11 +169,17 @@ static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type,
 ret = -1 - SIX_LOCK_write;
 }
 } else if (type == SIX_LOCK_write && lock->readers) {
- if (try) {
+ if (try)
 atomic_add(SIX_LOCK_HELD_write, &lock->state);
- smp_mb__after_atomic();

fs/bcachefs/six.c static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type, · line 182

Quoted 10 lines
fs/bcachefs/six.c +8 -1
- }
 
+ /*
+  * Make sure atomic_add happens before pcpu_read_count and
+  * six_set_bitmask in slow path happens before pcpu_read_count.
+  *
+  * Paired with the smp_mb() in read lock fast path (per-cpu mode)
+  * and the one before atomic_read in read unlock path.
+  */
+ smp_mb();

What is this barrier ordering?

Quoted 7 lines
Quoted 3 lines

This is only a change in the !try case, and we didn't do anything before checking pcpu_read_count() - except way back in six_lock_slowpath() where we set SIX_LOCK_HELD_write, but that does have a barrier.

Make sure six_set_bitmask(lock, SIX_LOCK_WAITING_read << type)

happens before pcpu_read_count()

that's in another thread, a memory barrier orders things in the same thread

author
#4 lore raw

2024年9月1日 11:10,Kent Overstreet <[email protected]> wrote:

Quoted 51 lines · Alan Huang
Quoted 49 lines
Quoted 48 lines

2024年9月1日 09:09,Kent Overstreet <[email protected]> wrote:

Quoted 29 lines · Alan Huang

If the reader acquires the read lock and then the writer enters the slow path, while the reader proceeds to the unlock path, the following scenario can occur without the change:

writer: pcpu_read_count(lock) return 1 (so __do_six_trylock will return 0)
reader: this_cpu_dec(*lock->readers)
reader: smp_mb()
reader: state = atomic_read(&lock->state) (there is no waiting flag set)
writer: six_set_bitmask()

then the writer will sleep forever.

---
fs/bcachefs/six.c | 12 +++++++++---

1 file changed, 9 insertions(+), 3 deletions(-)
fs/bcachefs/six.c +1 -2
index 3a494c5d1247..b3583c50ef04 100644
@@ -169,11 +169,17 @@ static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type,
 ret = -1 - SIX_LOCK_write;
 }
 } else if (type == SIX_LOCK_write && lock->readers) {
- if (try) {
+ if (try)
 atomic_add(SIX_LOCK_HELD_write, &lock->state);
- smp_mb__after_atomic();

fs/bcachefs/six.c static int __do_six_trylock(struct six_lock *lock, enum six_lock_type type, · line 182

Quoted 10 lines
fs/bcachefs/six.c +8 -1
- }
 
+ /*
+  * Make sure atomic_add happens before pcpu_read_count and
+  * six_set_bitmask in slow path happens before pcpu_read_count.
+  *
+  * Paired with the smp_mb() in read lock fast path (per-cpu mode)
+  * and the one before atomic_read in read unlock path.
+  */
+ smp_mb();

What is this barrier ordering?

This is only a change in the !try case, and we didn't do anything before checking pcpu_read_count() - except way back in six_lock_slowpath() where we set SIX_LOCK_HELD_write, but that does have a barrier.

Quoted 6 lines
Quoted 3 lines

Make sure six_set_bitmask(lock, SIX_LOCK_WAITING_read << type)

happens before pcpu_read_count()

that's in another thread, a memory barrier orders things in the same thread

six_set_bitmask(lock, SIX_LOCK_WAITING_read << type) is called before __do_six_trylock()