Skip to contentSkip to conversation

bcachefs: journal bug fixes

PATCH Superseded Brian Foster 2 participants 14 messages Aug 31, 2023 – Sep 12, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#1 lore raw

PATCH 1/3 bcachefs: restart journal reclaim thread on ro->rw transitions

In reply to Brian Foster

Commit c2d5ff36065a4 ("bcachefs: Start journal reclaim thread earlier") tweaked reclaim thread management to start a bit earlier in the mount sequence by moving the start call from __bch2_fs_read_write() to bch2_fs_journal_start(). This has the side effect of never starting the reclaim thread on a ro->rw transition, which can be observed by monitoring reclaim behavior via the journal_reclaim tracepoints. I.e. once an fs has remounted ro->rw, we only ever rely on direct reclaim from that point forward.

Since bch2_journal_reclaim_start() properly handles the case where the reclaim thread has already been created, restore the start call in the read-write helper. This allows the reclaim thread to start early when appropriate and also exit/restart on remounts or freeze cycles. In the latter case it may be possible to simply allow the task to freeze rather than destroy it, but for now just fix the immediate bug.

---
 fs/bcachefs/super.c | 4 ++++
 1 file changed, 4 insertions(+)
fs/bcachefs/super.c +4 -0
index 60424865980d..29cd71445a94 100644
@@ -421,6 +421,10 @@ static int __bch2_fs_read_write(struct bch_fs *c, bool early)
 		return ret;
 	}
 
+	ret = bch2_journal_reclaim_start(&c->journal);
+	if (ret)
+		goto err;
+
 	if (!early) {
 		ret = bch2_fs_read_write_late(c);
 		if (ret)
Signature · 1 line
2.41.0
#6 lore raw

PATCH 3/3 bcachefs: fix race between journal entry close and pin set

In reply to Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

This indicates that one or more dirty key cache keys still exist after the attempt to flush and quiesce the fs. The sequence that leads to this problem actually occurs on unfreeze (ro->rw), and looks something like the following:

- Task A begins a transaction commit and acquires journal_res for the current seq. This transaction intends to perform key cache insertion.
- Task B begins a bch2_journal_flush() via bch2_sync_fs(). This ends up in journal_entry_want_write(), which closes the current journal entry and drops the reference to the pin list created on entry open. The pin put pops the front of the journal via fast reclaim since the reference count has dropped to 0.
- Task A attempts to set the journal pin for the associated cached key, but bch2_journal_pin_set() skips the pin insert because the seq of the transaction reservation is behind the front of the pin list fifo.

The end result is that the pin associated with the cached key is not added, which prevents a subsequent reclaim from processing the key and thus leaves it dangling at freeze time. The fundamental cause of this problem is that the front of the journal is allowed to pop before a transaction with outstanding reservation on the associated journal seq is able to add a pin. The count for the pin list associated with the seq drops to zero and is prematurely reclaimed as a result.

The logical fix for this problem lies in how the journal buffer is managed in similar scenarios where the entry might have been closed before a transaction with outstanding reservations happens to be committed.

When a journal entry is opened, the current sequence number is bumped, the associated pin list is initialized with a reference count of 1, and the journal buffer reference count is bumped (via journal_state_inc()). When a journal reservation is acquired, the reservation also acquires a reference on the associated buffer. If the journal entry is closed in the meantime, it drops both the pin and buffer references held by the open entry, but the buffer still has references held by outstanding reservation. After the associated transaction commits, the reservation release drops the associated buffer references and the buffer is written out once the reference count has dropped to zero.

The fundamental problem here is that the lifecycle of the pin list reference held by an open journal entry is too short to cover the processing of transactions with outstanding reservations. The simplest way to address this is to expand the pin list reference to the lifecycle of the buffer vs. the shorter lifecycle of the open journal entry. This allows a journal entry to closer for whatever reason, but ensures the pin list for the seq with outstanding reservation cannot be popped and reclaimed before all outstanding reservations have been released.

Move the pin put from journal entry close to where final reference count processing of the journal buffer occurs. Create a duplicate helper to cover the case where the caller doesn't already hold the journal lock. This allows generic/390 to pass reliably.

---
 fs/bcachefs/journal.c |  4 +---
 fs/bcachefs/journal.h | 19 +++++++++++++++++--
 2 files changed, 18 insertions(+), 5 deletions(-)
fs/bcachefs/journal.c +1 -3
index 76ebcdb3e3b4..21f97d48931d 100644
@@ -195,13 +195,11 @@ static void __journal_entry_close(struct journal *j, unsigned closed_val)
 	buf->data->last_seq	= cpu_to_le64(buf->last_seq);
 	BUG_ON(buf->last_seq > le64_to_cpu(buf->data->seq));
 
-	__bch2_journal_pin_put(j, le64_to_cpu(buf->data->seq));
-
 	cancel_delayed_work(&j->write_work);
 
 	bch2_journal_space_available(j);
 
-	bch2_journal_buf_put(j, old.idx);
+	__bch2_journal_buf_put(j, old.idx, le64_to_cpu(buf->data->seq));
 }
 
 void bch2_journal_halt(struct journal *j)
fs/bcachefs/journal.h +17 -2
index c7a31da077c9..5dea5d46c616 100644
@@ -112,6 +112,7 @@
 #include <linux/hash.h>
 
 #include "journal_types.h"
+#include "journal_reclaim.h"
 
 struct bch_fs;
 
@@ -269,13 +270,27 @@ static inline union journal_res_state journal_state_buf_put(struct journal *j, u
 }
 
 void bch2_journal_write(struct closure *);
-static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+static inline void __bch2_journal_buf_put(struct journal *j, unsigned idx, u64 seq)
 {
 	struct bch_fs *c = container_of(j, struct bch_fs, journal);
 	union journal_res_state s;
 
 	s = journal_state_buf_put(j, idx);
 	if (!journal_state_count(s, idx)) {
+		__bch2_journal_pin_put(j, seq);
+		if (idx == s.unwritten_idx)
+			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
+	}
+}
+
+static inline void bch2_journal_buf_put(struct journal *j, unsigned idx, u64 seq)
+{
+	struct bch_fs *c = container_of(j, struct bch_fs, journal);
+	union journal_res_state s;
+
+	s = journal_state_buf_put(j, idx);
+	if (!journal_state_count(s, idx)) {
+		bch2_journal_pin_put(j, seq);
 		if (idx == s.unwritten_idx)
 			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
 	}
@@ -298,7 +313,7 @@ static inline void bch2_journal_res_put(struct journal *j,
 				       BCH_JSET_ENTRY_btree_keys,
 				       0, 0, 0);
 
-	bch2_journal_buf_put(j, res->idx);
+	bch2_journal_buf_put(j, res->idx, res->seq);
 
 	res->ref = 0;
 }
Signature · 1 line
2.41.0
#0 lore raw

cover bcachefs: journal bug fixes

Hi all,

This series includes a couple bug fixes that fell out of ongoing experimentation with freeze support. I think patch 1 is pretty self-explanatory, patch 2 is a prepatory patch, and patch 3 addresses a race in the journaling code.

All patches are available in my current test branch, with CI test results below [1]. Note that branch also includes a patch to enable freeze, but I'd rather not see that one land quite yet (which is why it's not included here). It's so far only seen the type of testing intended to shake these sorts of peripheral issues out, and I need to run some more testing to confirm sane behavior.

Thoughts, reviews, flames appreciated.

Brian

[1] https://evilpiepirate.org/~testdashboard/ci?branch=bfoster

Brian Foster (3):
bcachefs: restart journal reclaim thread on ro->rw transitions
bcachefs: prepare journal buf put to handle pin put
bcachefs: fix race between journal entry close and pin set

 fs/bcachefs/journal.c | 13 +------------
 fs/bcachefs/journal.h | 39 +++++++++++++++++++++++++++++++++------
 fs/bcachefs/super.c   |  4 ++++
 3 files changed, 38 insertions(+), 18 deletions(-)
Signature · 1 line
2.41.0
#2 lore raw

PATCH 2/3 bcachefs: prepare journal buf put to handle pin put

bcachefs freeze testing has uncovered some raciness between journal entry open/close and pin list reference count management. The details of the problem are described in a separate patch. In preparation for the associated fix, refactor the journal buffer put path a bit to allow it to eventually handle dropping the pin list reference currently held by an open journal entry. Open code the journal write dispatch since it is essentially a single line and instead factor out the journal state update to be reused, consistent with other journal state update helpers. No functional changes in this patch.

---
 fs/bcachefs/journal.c |  9 ---------
 fs/bcachefs/journal.h | 22 +++++++++++++++++-----
 2 files changed, 17 insertions(+), 14 deletions(-)
fs/bcachefs/journal.c +0 -9
index 055920c26da6..76ebcdb3e3b4 100644
@@ -132,15 +132,6 @@ journal_error_check_stuck(struct journal *j, int error, unsigned flags)
 	return stuck;
 }
 
-/* journal entry close/open: */
-
-void __bch2_journal_buf_put(struct journal *j)
-{
-	struct bch_fs *c = container_of(j, struct bch_fs, journal);
-
-	closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
-}
-
 /*
  * Returns true if journal entry is now closed:
  *
fs/bcachefs/journal.h +17 -5
index 008a2e25a4fa..c7a31da077c9 100644
@@ -252,9 +252,10 @@ static inline bool journal_entry_empty(struct jset *j)
 	return true;
 }
 
-void __bch2_journal_buf_put(struct journal *);
-
-static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+/*
+ * Drop reference on a buffer index and return true if the count has hit zero.
+ */
+static inline union journal_res_state journal_state_buf_put(struct journal *j, unsigned idx)
 {
 	union journal_res_state s;
 
@@ -264,9 +265,20 @@ static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
 				    .buf2_count = idx == 2,
 				    .buf3_count = idx == 3,
 				    }).v, &j->reservations.counter);
+	return s;
+}
 
-	if (!journal_state_count(s, idx) && idx == s.unwritten_idx)
-		__bch2_journal_buf_put(j);
+void bch2_journal_write(struct closure *);
+static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+{
+	struct bch_fs *c = container_of(j, struct bch_fs, journal);
+	union journal_res_state s;
+
+	s = journal_state_buf_put(j, idx);
+	if (!journal_state_count(s, idx)) {
+		if (idx == s.unwritten_idx)
+			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
+	}
 }
 
 /*
Signature · 1 line
2.41.0
#3 lore raw

Re: [PATCH 2/3] bcachefs: prepare journal buf put to handle pin put

Quoted 10 lines · Brian Foster

bcachefs freeze testing has uncovered some raciness between journal entry open/close and pin list reference count management. The details of the problem are described in a separate patch. In preparation for the associated fix, refactor the journal buffer put path a bit to allow it to eventually handle dropping the pin list reference currently held by an open journal entry. Open code the journal write dispatch since it is essentially a single line and instead factor out the journal state update to be reused, consistent with other journal state update helpers. No functional changes in this patch.

So, I don't want to take this patch as is because even though __bch2_journal_buf_put() is small, you're putting it into an inline fastpath that we want to be as small and fast as possible - the main transaction commit path.

journal_state_buf_put() is fine though, happy to take that part of the patch.

Quoted 71 lines
---
 fs/bcachefs/journal.c |  9 ---------
 fs/bcachefs/journal.h | 22 +++++++++++++++++-----
 2 files changed, 17 insertions(+), 14 deletions(-)
fs/bcachefs/journal.c +0 -9
index 055920c26da6..76ebcdb3e3b4 100644
@@ -132,15 +132,6 @@ journal_error_check_stuck(struct journal *j, int error, unsigned flags)
 	return stuck;
 }
 
-/* journal entry close/open: */
-
-void __bch2_journal_buf_put(struct journal *j)
-{
-	struct bch_fs *c = container_of(j, struct bch_fs, journal);
-
-	closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
-}
-
 /*
  * Returns true if journal entry is now closed:
  *
fs/bcachefs/journal.h +17 -5
index 008a2e25a4fa..c7a31da077c9 100644
@@ -252,9 +252,10 @@ static inline bool journal_entry_empty(struct jset *j)
 	return true;
 }
 
-void __bch2_journal_buf_put(struct journal *);
-
-static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+/*
+ * Drop reference on a buffer index and return true if the count has hit zero.
+ */
+static inline union journal_res_state journal_state_buf_put(struct journal *j, unsigned idx)
 {
 	union journal_res_state s;
 
@@ -264,9 +265,20 @@ static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
 				    .buf2_count = idx == 2,
 				    .buf3_count = idx == 3,
 				    }).v, &j->reservations.counter);
+	return s;
+}
 
-	if (!journal_state_count(s, idx) && idx == s.unwritten_idx)
-		__bch2_journal_buf_put(j);
+void bch2_journal_write(struct closure *);
+static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+{
+	struct bch_fs *c = container_of(j, struct bch_fs, journal);
+	union journal_res_state s;
+
+	s = journal_state_buf_put(j, idx);
+	if (!journal_state_count(s, idx)) {
+		if (idx == s.unwritten_idx)
+			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
+	}
 }
 
 /*
Signature · 1 line
2.41.0
#7 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

In reply to Brian Foster
Quoted 59 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

This indicates that one or more dirty key cache keys still exist after the attempt to flush and quiesce the fs. The sequence that leads to this problem actually occurs on unfreeze (ro->rw), and looks something like the following:

- Task A begins a transaction commit and acquires journal_res for the current seq. This transaction intends to perform key cache insertion.
- Task B begins a bch2_journal_flush() via bch2_sync_fs(). This ends up in journal_entry_want_write(), which closes the current journal entry and drops the reference to the pin list created on entry open. The pin put pops the front of the journal via fast reclaim since the reference count has dropped to 0.
- Task A attempts to set the journal pin for the associated cached key, but bch2_journal_pin_set() skips the pin insert because the seq of the transaction reservation is behind the front of the pin list fifo.

The end result is that the pin associated with the cached key is not added, which prevents a subsequent reclaim from processing the key and thus leaves it dangling at freeze time. The fundamental cause of this problem is that the front of the journal is allowed to pop before a transaction with outstanding reservation on the associated journal seq is able to add a pin. The count for the pin list associated with the seq drops to zero and is prematurely reclaimed as a result.

The logical fix for this problem lies in how the journal buffer is managed in similar scenarios where the entry might have been closed before a transaction with outstanding reservations happens to be committed.

When a journal entry is opened, the current sequence number is bumped, the associated pin list is initialized with a reference count of 1, and the journal buffer reference count is bumped (via journal_state_inc()). When a journal reservation is acquired, the reservation also acquires a reference on the associated buffer. If the journal entry is closed in the meantime, it drops both the pin and buffer references held by the open entry, but the buffer still has references held by outstanding reservation. After the associated transaction commits, the reservation release drops the associated buffer references and the buffer is written out once the reference count has dropped to zero.

The fundamental problem here is that the lifecycle of the pin list reference held by an open journal entry is too short to cover the processing of transactions with outstanding reservations. The simplest way to address this is to expand the pin list reference to the lifecycle of the buffer vs. the shorter lifecycle of the open journal entry. This allows a journal entry to closer for whatever reason, but ensures the pin list for the seq with outstanding reservation cannot be popped and reclaimed before all outstanding reservations have been released.

Quoted 6 lines

Move the pin put from journal entry close to where final reference count processing of the journal buffer occurs. Create a duplicate helper to cover the case where the caller doesn't already hold the journal lock. This allows generic/390 to pass reliably.

Good find :)

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

We could just change it to check if the pin list is for a journal buf that's still open and has a nonzero refcount.

It seems on the surface to be a bug where just fixing the lifetime refcounts is the right approach, but I think that ends up being the trickier approach here, since bch2_journal_buf_put() now has a couple things it's checking at different stages of the journal buf lifetime.

#8 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

What do you think of this fix?

fs/bcachefs/journal_reclaim.c +6 -0
index 10e1860dad..021c56ac67 100644
@@ -303,6 +303,12 @@ static void bch2_journal_reclaim_fast(struct journal *j)
 	 */
 	while (!fifo_empty(&j->pin) &&
 	       !atomic_read(&fifo_peek_front(&j->pin).count)) {
+		u64 seq = journal_last_seq(j);
+
+		if (seq > j->seq_ondisk &&
+		    journal_state_count(j->reservations, seq & JOURNAL_BUF_MASK))
+			break;
+
 		fifo_pop(&j->pin, temp);
 		popped = true;
 	}
#9 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

Quoted 20 lines · Kent Overstreet
Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

What do you think of this fix?

fs/bcachefs/journal_reclaim.c +0 -0
index 10e1860dad..021c56ac67 100644
@@ -303,6 +303,12 @@ static void bch2_journal_reclaim_fast(struct journal *j)
 	 */
 	while (!fifo_empty(&j->pin) &&

fs/bcachefs/journal_reclaim.c static void bch2_journal_reclaim_fast(struct journal *j) · line 314

Quoted 10 lines
fs/bcachefs/journal_reclaim.c +6 -0
 	       !atomic_read(&fifo_peek_front(&j->pin).count)) {
+		u64 seq = journal_last_seq(j);
+
+		if (seq > j->seq_ondisk &&
+		    journal_state_count(j->reservations, seq & JOURNAL_BUF_MASK))
+			break;
+
 		fifo_pop(&j->pin, temp);
 		popped = true;
 	}

So my first reaction once I figured out what was going on was to consider something like this, because it seemed like the issue was premature release of the journal tail. After digging into that a bit it didn't quite feel right, which is what had me digging further into why the ref count was zero on a seq with outstanding reservation.

I think I see what the above is doing, but I really don't see how this is simplified in any way. Maybe the code changes are slightly smaller, but IMO the bigger picture is far easier to reason about when the reference counts are properly honored. I.e., the outstanding reservations hold references on the journal buffer associated with the seq, and the active journal buffer holds a reference on the pin fifo.

Conversely, the above strikes me as unnecessarily clever based on the fact that just to review it I need to grok behavior in several different contexts (res and seq management, on disk seq updates, etc.) to convince myself whether it's safe, and even then I'm not totally sure I have enough context to say whether the state count check can't ever break on something like a reused (for a new seq) journal buffer. Heh, the more I think about that the more I have to ask: why not just make the reference count work? ;P

FWIW, one thing I disliked about the original patch was the need to mostly duplicate the buf put helper due to the locking context quirk. I was trying to avoid that, but I ran out of ideas before I wanted to move on actually fixing the bug. My preference would be to address the reference counting issue as is (to preserve design simplicity), and then maybe think a bit harder about cleaning up the res put implementation if the primary concern is that we feel like this starts to make things a bit convoluted..

Brian

#4 lore raw

Re: [PATCH 2/3] bcachefs: prepare journal buf put to handle pin put

In reply to Kent Overstreet
Quoted 12 lines · Kent Overstreet
Quoted 10 lines · Brian Foster

bcachefs freeze testing has uncovered some raciness between journal entry open/close and pin list reference count management. The details of the problem are described in a separate patch. In preparation for the associated fix, refactor the journal buffer put path a bit to allow it to eventually handle dropping the pin list reference currently held by an open journal entry. Open code the journal write dispatch since it is essentially a single line and instead factor out the journal state update to be reused, consistent with other journal state update helpers. No functional changes in this patch.

Quoted 5 lines

So, I don't want to take this patch as is because even though __bch2_journal_buf_put() is small, you're putting it into an inline fastpath that we want to be as small and fast as possible - the main transaction commit path.

Can you elaborate on what you mean here? ISTM all this patch does is replace a function call to __bch2_journal_buf_put() that already exists..

Brian

Quoted 75 lines

journal_state_buf_put() is fine though, happy to take that part of the patch.

Quoted 71 lines
---
 fs/bcachefs/journal.c |  9 ---------
 fs/bcachefs/journal.h | 22 +++++++++++++++++-----
 2 files changed, 17 insertions(+), 14 deletions(-)
fs/bcachefs/journal.c +0 -9
index 055920c26da6..76ebcdb3e3b4 100644
@@ -132,15 +132,6 @@ journal_error_check_stuck(struct journal *j, int error, unsigned flags)
 	return stuck;
 }
 
-/* journal entry close/open: */
-
-void __bch2_journal_buf_put(struct journal *j)
-{
-	struct bch_fs *c = container_of(j, struct bch_fs, journal);
-
-	closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
-}
-
 /*
  * Returns true if journal entry is now closed:
  *
fs/bcachefs/journal.h +17 -5
index 008a2e25a4fa..c7a31da077c9 100644
@@ -252,9 +252,10 @@ static inline bool journal_entry_empty(struct jset *j)
 	return true;
 }
 
-void __bch2_journal_buf_put(struct journal *);
-
-static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+/*
+ * Drop reference on a buffer index and return true if the count has hit zero.
+ */
+static inline union journal_res_state journal_state_buf_put(struct journal *j, unsigned idx)
 {
 	union journal_res_state s;
 
@@ -264,9 +265,20 @@ static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
 				    .buf2_count = idx == 2,
 				    .buf3_count = idx == 3,
 				    }).v, &j->reservations.counter);
+	return s;
+}
 
-	if (!journal_state_count(s, idx) && idx == s.unwritten_idx)
-		__bch2_journal_buf_put(j);
+void bch2_journal_write(struct closure *);
+static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+{
+	struct bch_fs *c = container_of(j, struct bch_fs, journal);
+	union journal_res_state s;
+
+	s = journal_state_buf_put(j, idx);
+	if (!journal_state_count(s, idx)) {
+		if (idx == s.unwritten_idx)
+			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
+	}
 }
 
 /*
Signature · 1 line
2.41.0
#10 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

In reply to Brian Foster
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

...

Quoted 10 lines

FWIW, one thing I disliked about the original patch was the need to mostly duplicate the buf put helper due to the locking context quirk. I was trying to avoid that, but I ran out of ideas before I wanted to move on actually fixing the bug. My preference would be to address the reference counting issue as is (to preserve design simplicity), and then maybe think a bit harder about cleaning up the res put implementation if the primary concern is that we feel like this starts to make things a bit convoluted..

Another thought that comes to mind is to perhaps allow the journal_res to hold a reference to the pin fifo for the associated seq.. The idea would be we could continue to hold a reference during the open/close journal buffer lifecycle, but a res of the same seq would acquire an additional reference as well to keep the tail from popping before a transaction can actually set a pin (i.e. essentially parallel to the buffer reference).

In a sense the behavior in this patch is kind of an optimization of that approach, but I think implementing it directly would eliminate the need to mostly duplicate the put helper, perhaps making things a bit more natural. We could still fall into bch2_journal_reclaim_fast() from res_put(), but now only for cases where the reservation outlives the active journal buffer (due to a journal flush or whatever). That also still addresses the race by properly using the reference count. I'd probably have to try it to be sure I'm not missing something, but... thoughts?

Brian

#5 lore raw

Re: [PATCH 2/3] bcachefs: prepare journal buf put to handle pin put

In reply to Brian Foster
Quoted 19 lines · Brian Foster
Quoted 12 lines · Kent Overstreet
Quoted 10 lines · Brian Foster

bcachefs freeze testing has uncovered some raciness between journal entry open/close and pin list reference count management. The details of the problem are described in a separate patch. In preparation for the associated fix, refactor the journal buffer put path a bit to allow it to eventually handle dropping the pin list reference currently held by an open journal entry. Open code the journal write dispatch since it is essentially a single line and instead factor out the journal state update to be reused, consistent with other journal state update helpers. No functional changes in this patch.

Quoted 5 lines

So, I don't want to take this patch as is because even though __bch2_journal_buf_put() is small, you're putting it into an inline fastpath that we want to be as small and fast as possible - the main transaction commit path.

Quoted 3 lines

Can you elaborate on what you mean here? ISTM all this patch does is replace a function call to __bch2_journal_buf_put() that already exists..

Yes, you made the code in __bch2_journal_buf_put() inline. Since that's the slowpath, we don't want it inline...

Quoted 79 lines

Brian

Quoted 75 lines

journal_state_buf_put() is fine though, happy to take that part of the patch.

Quoted 71 lines
---
 fs/bcachefs/journal.c |  9 ---------
 fs/bcachefs/journal.h | 22 +++++++++++++++++-----
 2 files changed, 17 insertions(+), 14 deletions(-)
fs/bcachefs/journal.c +0 -9
index 055920c26da6..76ebcdb3e3b4 100644
@@ -132,15 +132,6 @@ journal_error_check_stuck(struct journal *j, int error, unsigned flags)
 	return stuck;
 }
 
-/* journal entry close/open: */
-
-void __bch2_journal_buf_put(struct journal *j)
-{
-	struct bch_fs *c = container_of(j, struct bch_fs, journal);
-
-	closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
-}
-
 /*
  * Returns true if journal entry is now closed:
  *
fs/bcachefs/journal.h +17 -5
index 008a2e25a4fa..c7a31da077c9 100644
@@ -252,9 +252,10 @@ static inline bool journal_entry_empty(struct jset *j)
 	return true;
 }
 
-void __bch2_journal_buf_put(struct journal *);
-
-static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+/*
+ * Drop reference on a buffer index and return true if the count has hit zero.
+ */
+static inline union journal_res_state journal_state_buf_put(struct journal *j, unsigned idx)
 {
 	union journal_res_state s;
 
@@ -264,9 +265,20 @@ static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
 				    .buf2_count = idx == 2,
 				    .buf3_count = idx == 3,
 				    }).v, &j->reservations.counter);
+	return s;
+}
 
-	if (!journal_state_count(s, idx) && idx == s.unwritten_idx)
-		__bch2_journal_buf_put(j);
+void bch2_journal_write(struct closure *);
+static inline void bch2_journal_buf_put(struct journal *j, unsigned idx)
+{
+	struct bch_fs *c = container_of(j, struct bch_fs, journal);
+	union journal_res_state s;
+
+	s = journal_state_buf_put(j, idx);
+	if (!journal_state_count(s, idx)) {
+		if (idx == s.unwritten_idx)
+			closure_call(&j->io, bch2_journal_write, c->io_complete_wq, NULL);
+	}
 }
 
 /*
Signature · 1 line
2.41.0
#11 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

In reply to Brian Foster
Quoted 23 lines · Brian Foster
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

...

Quoted 10 lines

FWIW, one thing I disliked about the original patch was the need to mostly duplicate the buf put helper due to the locking context quirk. I was trying to avoid that, but I ran out of ideas before I wanted to move on actually fixing the bug. My preference would be to address the reference counting issue as is (to preserve design simplicity), and then maybe think a bit harder about cleaning up the res put implementation if the primary concern is that we feel like this starts to make things a bit convoluted..

Quoted 7 lines

Another thought that comes to mind is to perhaps allow the journal_res to hold a reference to the pin fifo for the associated seq.. The idea would be we could continue to hold a reference during the open/close journal buffer lifecycle, but a res of the same seq would acquire an additional reference as well to keep the tail from popping before a transaction can actually set a pin (i.e. essentially parallel to the buffer reference).

Yeah that would probably be cleanest - but it would be much too heavy.

Quoted 9 lines

In a sense the behavior in this patch is kind of an optimization of that approach, but I think implementing it directly would eliminate the need to mostly duplicate the put helper, perhaps making things a bit more natural. We could still fall into bch2_journal_reclaim_fast() from res_put(), but now only for cases where the reservation outlives the active journal buffer (due to a journal flush or whatever). That also still addresses the race by properly using the reference count. I'd probably have to try it to be sure I'm not missing something, but... thoughts?

We can do your approach if that's what you feel is going to be cleanest, but if we go that way here's my two main concerns:

- we can't put more slowpath code into the inline fastpath, remember this is all inlined into the main __bch2_trans_commit() path

- your approach (the optimized version) means there's a new state transition where we do work, i.e. when journal_state_count() hits 0.

So we have to make sure that that refcount isn't hitting 0 multiple times. The appropriate assertion already exists in journal_res_get_fast() - for a refcount to hit 0 multiple times it must leave 0 while in use, and we check for that.

Let's just add a comment by that EBUG_ON(!journal_state_count(new, new.idx)) line indicating why it's now important.

#12 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

Quoted 32 lines · Kent Overstreet
Quoted 23 lines · Brian Foster
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

...

Quoted 10 lines

FWIW, one thing I disliked about the original patch was the need to mostly duplicate the buf put helper due to the locking context quirk. I was trying to avoid that, but I ran out of ideas before I wanted to move on actually fixing the bug. My preference would be to address the reference counting issue as is (to preserve design simplicity), and then maybe think a bit harder about cleaning up the res put implementation if the primary concern is that we feel like this starts to make things a bit convoluted..

Quoted 7 lines

Another thought that comes to mind is to perhaps allow the journal_res to hold a reference to the pin fifo for the associated seq.. The idea would be we could continue to hold a reference during the open/close journal buffer lifecycle, but a res of the same seq would acquire an additional reference as well to keep the tail from popping before a transaction can actually set a pin (i.e. essentially parallel to the buffer reference).

Quoted 2 lines

Yeah that would probably be cleanest - but it would be much too heavy.

I'm curious how so? Additional atomics?

If so, we could also consider something that allows a pin list ref on the reservation to simply transfer to a pin set via the transaction. That might not add any new overhead over the current code, but would require some plumbing.

Quoted 10 lines
Quoted 9 lines

In a sense the behavior in this patch is kind of an optimization of that approach, but I think implementing it directly would eliminate the need to mostly duplicate the put helper, perhaps making things a bit more natural. We could still fall into bch2_journal_reclaim_fast() from res_put(), but now only for cases where the reservation outlives the active journal buffer (due to a journal flush or whatever). That also still addresses the race by properly using the reference count. I'd probably have to try it to be sure I'm not missing something, but... thoughts?

Quoted 3 lines

We can do your approach if that's what you feel is going to be cleanest, but if we go that way here's my two main concerns:

Ok..

FWIW, it's not so much that I think it's the cleanest (I agree that the variant discussed above probably ends up cleaner code-wise), but rather that it provides sufficient confidence we won't just have to revisit the same underlying problem the next time some bit of code assumes that an outstanding reservation guarantees a usable pin list.

Quoted 3 lines

- we can't put more slowpath code into the inline fastpath, remember this is all inlined into the main __bch2_trans_commit() path

Sure. I missed that closure_call() was itself inlined. I think that is what initially confused me wrt to the feedback on patch 2. I'll try to keep that helper external and perhaps just rename it.

Quoted 3 lines

- your approach (the optimized version) means there's a new state transition where we do work, i.e. when journal_state_count() hits 0.

Indeed.

Quoted 8 lines

So we have to make sure that that refcount isn't hitting 0 multiple times. The appropriate assertion already exists in journal_res_get_fast() - for a refcount to hit 0 multiple times it must leave 0 while in use, and we check for that.

Let's just add a comment by that EBUG_ON(!journal_state_count(new, new.idx)) line indicating why it's now important.

Sounds reasonable. I'll try to come up with something. Thanks.

Brian

#13 lore raw

Re: [PATCH 3/3] bcachefs: fix race between journal entry close and pin set

Quoted 33 lines · Brian Foster
Quoted 32 lines · Kent Overstreet
Quoted 30 lines · Brian Foster
Quoted 10 lines · Brian Foster
Quoted 9 lines · Kent Overstreet
Quoted 8 lines · Kent Overstreet
Quoted 4 lines · Brian Foster

bcachefs freeze testing via fstests generic/390 occasionally reproduces the following BUG from bch2_fs_read_only():

  BUG_ON(atomic_long_read(&c->btree_key_cache.nr_dirty));

Your fix makes sense, but I wonder if it might be simpler and better to fix this in bch2_journal_reclaim_fast().

...

Quoted 10 lines

FWIW, one thing I disliked about the original patch was the need to mostly duplicate the buf put helper due to the locking context quirk. I was trying to avoid that, but I ran out of ideas before I wanted to move on actually fixing the bug. My preference would be to address the reference counting issue as is (to preserve design simplicity), and then maybe think a bit harder about cleaning up the res put implementation if the primary concern is that we feel like this starts to make things a bit convoluted..

Another thought that comes to mind is to perhaps allow the journal_res to hold a reference to the pin fifo for the associated seq.. The idea would be we could continue to hold a reference during the open/close journal buffer lifecycle, but a res of the same seq would acquire an additional reference as well to keep the tail from popping before a transaction can actually set a pin (i.e. essentially parallel to the buffer reference).

Quoted 4 lines
Quoted 2 lines

Yeah that would probably be cleanest - but it would be much too heavy.

I'm curious how so? Additional atomics?

Not just that, getting a journal reservation is lockless but getting a journal pin is not.

Quoted 10 lines
Quoted 8 lines

So we have to make sure that that refcount isn't hitting 0 multiple times. The appropriate assertion already exists in journal_res_get_fast() - for a refcount to hit 0 multiple times it must leave 0 while in use, and we check for that.

Let's just add a comment by that EBUG_ON(!journal_state_count(new, new.idx)) line indicating why it's now important.

Sounds reasonable. I'll try to come up with something. Thanks.

Sounds good