Skip to content

btree write buffer & journal optimizations

Kent Overstreet linux-bcachefs 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest17 patches 11 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/17bcachefs: Kill journal pre-reservationsfs/bcachefs/btree_iter.c fs/bcachefs/btree_key_cache.c fs/bcachefs/btree_trans_commit.c +8 more+19 -2700 replies
2/17bcachefs: track_event_change()fs/bcachefs/bcachefs.h fs/bcachefs/journal.c fs/bcachefs/journal_io.c +5 more+140 -920 replies
3/17bcachefs: Journal pins must always have a flush_fnfs/bcachefs/btree_trans_commit.c fs/bcachefs/btree_update_interior.c fs/bcachefs/btree_write_buffer.c +1 more+40 -203 replies
4/17bcachefs: BTREE_INSERT_JOURNAL_REPLAY now "don't init trans->journal_res"fs/bcachefs/btree_trans_commit.c fs/bcachefs/recovery.c+17 -40 replies
5/17bcachefs: Kill BTREE_UPDATE_PREJOURNALfs/bcachefs/bkey_methods.h fs/bcachefs/btree_trans_commit.c fs/bcachefs/btree_types.h +2 more+11 -363 replies
6/17bcachefs: Go rw before journal replayfs/bcachefs/recovery.c+12 -40 replies
7/17bcachefs: Make journal replay more efficientfs/bcachefs/errcode.h fs/bcachefs/recovery.c+56 -292 replies
8/17bcachefs: Don't flush journal after replayfs/bcachefs/recovery.c+1 -30 replies
9/17bcachefs: Unwritten journal buffers are always dirtyfs/bcachefs/journal_io.c fs/bcachefs/journal_reclaim.c+2 -00 replies
10/17bcachefs: journal->buf_lockfs/bcachefs/journal.c fs/bcachefs/journal_io.c fs/bcachefs/journal_types.h+9 -00 replies
11/17bcachefs: bch2_journal_block_reservations()fs/bcachefs/journal.c fs/bcachefs/journal.h+31 -10 replies
12/17bcachefs: Clean up btree write buffer write ref handlingfs/bcachefs/bcachefs.h fs/bcachefs/btree_trans_commit.c fs/bcachefs/btree_write_buffer.c +2 more+32 -220 replies
13/17bcachefs: bch2_btree_write_buffer_flush_locked()fs/bcachefs/btree_trans_commit.c fs/bcachefs/btree_write_buffer.c fs/bcachefs/btree_write_buffer.h+18 -130 replies
14/17bcachefs: bch2_btree_write_buffer_flush() -> bch2_btree_write_buffer_tryflush()fs/bcachefs/alloc_background.c fs/bcachefs/btree_write_buffer.c fs/bcachefs/btree_write_buffer.h +3 more+9 -100 replies
15/17bcachefs: Improve btree write buffer tracepointsfs/bcachefs/btree_write_buffer.c fs/bcachefs/trace.h+27 -80 replies
16/17bcachefs: btree write buffer now slurps keys from journalfs/bcachefs/bcachefs.h fs/bcachefs/bcachefs_format.h fs/bcachefs/btree_trans_commit.c +11 more+512 -2582 replies
17/17bcachefs: Inline btree write buffer sortfs/bcachefs/btree_write_buffer.c+48 -90 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/17] btree write buffer & journal optimizations

Large stack of changes - performance optimizations and improvements to related code.

The first big one, killing journal pre-reservations, is already merged:
besides being a performance improvement it turned out to fix a few tests that were sporadically failing with "journal stuck" - nice. The idea with that patch is that instead of reserving space in the journal ahead of time (for journal reservations that must succeed or risk deadlock), we instead rely on watermarks and tracking which operations are necessary for reclaim - when we're low on space in the journal and flushing things for reclaim, we're able to flush things in order, or track when we're not flushing things in order and allow it to fail

Most of the other patches are prep work leading up to the big btree write buffer rewrite, which changes the btree write buffer to pull keys from the journal, instead of having the tranaction commit path directly add keys to the btree write buffer.

btree write buffer performance is pretty important for multithreaded update workloads - it's an amdahl's law situation, flushing it is single threaded. More performance optimizations there would still be worthwhile, and I've been looking at it because I'm sketching out a rewrite of disk space accounting (which will allow for per snapshot Id accounting) that will lean heavily on the btree write buffer.

Kent Overstreet (17):
bcachefs: Kill journal pre-reservations
bcachefs: track_event_change()
bcachefs: Journal pins must always have a flush_fn
bcachefs: BTREE_INSERT_JOURNAL_REPLAY now "don't init

    trans->journal_res"

bcachefs: Kill BTREE_UPDATE_PREJOURNAL
bcachefs: Go rw before journal replay
bcachefs: Make journal replay more efficient
bcachefs: Don't flush journal after replay
bcachefs: Unwritten journal buffers are always dirty
bcachefs: journal->buf_lock
bcachefs: bch2_journal_block_reservations()
bcachefs: Clean up btree write buffer write ref handling
bcachefs: bch2_btree_write_buffer_flush_locked()
bcachefs: bch2_btree_write_buffer_flush() ->
bch2_btree_write_buffer_tryflush()
bcachefs: Improve btree write buffer tracepoints
bcachefs: btree write buffer now slurps keys from journal
bcachefs: Inline btree write buffer sort

 fs/bcachefs/alloc_background.c         |   2 +-
 fs/bcachefs/bcachefs.h                 |  18 +-
 fs/bcachefs/bcachefs_format.h          |   3 +-
 fs/bcachefs/bkey_methods.h             |   2 -
 fs/bcachefs/btree_iter.c               |   2 -
 fs/bcachefs/btree_key_cache.c          |  14 -
 fs/bcachefs/btree_trans_commit.c       | 123 ++----
 fs/bcachefs/btree_types.h              |   4 -
 fs/bcachefs/btree_update.c             |  23 -
 fs/bcachefs/btree_update_interior.c    |  46 +-
 fs/bcachefs/btree_update_interior.h    |   1 -
 fs/bcachefs/btree_write_buffer.c       | 584 +++++++++++++++++--------
 fs/bcachefs/btree_write_buffer.h       |  48 +-
 fs/bcachefs/btree_write_buffer_types.h |  37 +-
 fs/bcachefs/ec.c                       |   2 +-
 fs/bcachefs/errcode.h                  |   2 -
 fs/bcachefs/inode.c                    |  10 +-
 fs/bcachefs/journal.c                  | 120 +++--
 fs/bcachefs/journal.h                  | 102 +----
 fs/bcachefs/journal_io.c               |  57 ++-
 fs/bcachefs/journal_reclaim.c          |  66 +--
 fs/bcachefs/journal_reclaim.h          |   1 +
 fs/bcachefs/journal_types.h            |  40 +-
 fs/bcachefs/move.c                     |   7 +-
 fs/bcachefs/movinggc.c                 |   4 +-
 fs/bcachefs/recovery.c                 | 106 +++--
 fs/bcachefs/super.c                    |   6 +-
 fs/bcachefs/trace.h                    |  42 +-
 fs/bcachefs/util.c                     | 140 +++---
 fs/bcachefs/util.h                     |  33 +-
 30 files changed, 925 insertions(+), 720 deletions(-)
Signature · 1 line
2.42.0